:W&BᾄÜu´M 1ussyakcª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0°^i˜„‚" jx4sthf4j21i 1ussyakcŠà 1ussyakcescheuller-uc-san-diego spatop-sweep"  _wandb‚{}  batch_size‚2048  dropout‚0.13015882487632813  hidden_dim‚256 ( learning_rate‚0.00027406640964131203  num_embedding_layers‚10Bdpfx_vanilla_1ussyakcbnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ ‚žêÒÀÎýš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64)G?¨; 1ussyakc*  _wandb‚ {"runtime":0}  _runtime‚0†ÿq÷bÒÜ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 žêÒðÀò*-of*>/data/spatop/options_files/sweep_fixed_dp/config_1ussyakc.json*-l* /scratch/logs*-n*dpfx_vanilla_1ussyakc2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº 5m99h66c0ej28my30d8a2w9ld1xwssbqû5ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ Ôëúª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º 5m99h66c0ej28my30d8a2w9ld1xwssbqµª½Ç2  requirements.txtôÆáFW 1ussyakcZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64á@æÜY 1ussyakcZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64V Ö*  1ussyakcb trainer/global_step2õÂl&\  1ussyakcZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64VT/  1ussyakcb*"trainer/global_step2_ÑE.K  1ussyakcj: ‚žêÒØàÉÚ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] t^²ã“  1ussyakcj  ‚žêÒˆõÕÏî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 5.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 75.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ÄÒ0\ 1ussyakcjÊ ‚žêÒ¨ÉëзTrainable params: 84.5 M Non-trainable params: 50 Total params: 84.5 M Total estimated model params size (MB): 337.922 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 IŠ½:q 1ussyakcj` ‚žêÒ˜µ«ÑPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). G¤Úû{ 1ussyakcjj ‚žêÒ˜Ž¸ÑZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ?ÄÏq 1ussyakcj` ‚žêÒ ùËÑPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). Òôw+t 1ussyakcjc ‚žêÒ€¹ÖÑSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). B>un 1ussyakcj] ‚žêÒ˜áÑMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ¾Ã<õð 1ussyakc*Þ  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"}O<ë 1ussyakcjÙ ‚žêÒÈæ°ÖÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. 60UËN 1ussyakcj¼ ‚žêÒȘÊÖ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. p9ê 1ussyakcjØ ‡žêÒ óà*È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ƒ–Å O 1ussyakcj½ ‡žêÒˆÂó*­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. #8궜‚(:’ ‘žêҞ嘮 gpu.0.gpu‚98.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.74421691894531* gpu.0.memoryAllocatedBytes‚ 49601904640 gpu.0.temp‚41.0 gpu.0.powerWatts‚282.166' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.43319999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…©NÁÔ‚(:Ê ‘žêÒà ë¶ disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 154.40234375# proc.memory.rssMB‚ 2760.89453125* proc.memory.percent‚16.851162910461426$ disk./.usageGB‚574.7061080932617" disk.nvme0c0n1.in‚ 154.40234375 disk.nvme0n1.in‚ 154.40234375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚0.890625" disk.nvme0n1.out‚ 4.30908203125 disk.sda.out‚0.890625 disk.sda2.in‚0 disk.rbd2.out‚0 disk.drbd2165.out‚0$ memory_percent‚81.61380290985107$ disk.nvme0c0n1.out‚ 4.30908203125 disk.sda1.in‚0 cpu‚26.184831049981604 network.recv‚41169 disk.rbd1.out‚ 75.546875 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚32975 disk.sda3.out‚0.890625) proc.memory.availableMB‚ 3012.39453125) disk./.usagePercent‚38.56900912468198 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 4.30908203125 disk.dm-1.in‚0 disk.sda.in‚0Ø7,’‚(:ˆ  žêÒ¬ûÒ­ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.74665832519531* gpu.0.memoryAllocatedBytes‚ 49604001792 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èo«Ð‚(:Æ  žêÒùí® network.recv‚45153 disk.rbd1.out‚272.6875 cpu‚25.978092589539827 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 156.89453125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 156.89453125 disk.rbd1.in‚0$ memory_percent‚81.61582946777344% proc.memory.availableMB‚ 3012.0625) disk./.usagePercent‚38.56908131880663 disk.dm-0.in‚ 156.89453125 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 16.708984375 disk.sda2.out‚0 network.sent‚43421$ disk./.usageGB‚574.7071838378906 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.out‚2.09375 disk.sda.out‚2.09375 disk.rbd0.in‚0! proc.memory.rssMB‚ 2760.921875 disk.rbd2.out‚ 0.29296875 disk.dm-0.out‚ 16.708984375 disk.dm-1.out‚2.09375 disk.sda.in‚0 disk.sda2.in‚0* proc.memory.percent‚16.851329803466797 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0! disk.nvme0n1.out‚ 16.708984375 proc.cpu.threads‚17.DjÜ‚(:Ò ¯žêÒ ÿí disk.dm-1.in‚0 network.sent‚49010 disk.sda3.in‚0 disk.rbd1.out‚ 351.015625 disk.rbd2.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 41.6572265625 disk.sda2.in‚0 disk.sda3.out‚ 5.39453125 disk.rbd0.out‚0 disk.dm-0.in‚ 159.55859375 disk.drbd2165.in‚0$ memory_percent‚81.61888122558594 cpu‚25.84776755553104 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 159.55859375 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 41.6572265625) disk./.usagePercent‚38.56984498928127 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2760.9453125 disk.rbd2.out‚ 0.8046875% proc.memory.availableMB‚ 3011.5625* proc.memory.percent‚16.851472854614258# disk./.usageGB‚574.718563079834$ disk.nvme0c0n1.out‚ 41.6572265625 disk.sda1.in‚0 disk.dm-1.out‚ 5.39453125 network.recv‚49503 disk.nvme0n1.in‚ 159.55859375 disk.sda.out‚ 5.39453125 disk.sda2.out‚0=Ñ~‘‚(:‡ ¯žêÒÈ›ÉÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.749099731445305* gpu.0.memoryAllocatedBytes‚ 49606098944 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž ´œ‚(:’ ¾žêÒýÎЬ gpu.0.gpu‚87.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚43.0 gpu.0.powerWatts‚337.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.45760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CLŸÜ ‚(:Ò ¾žêÒأ㮠disk.drbd2165.out‚0) disk./.usagePercent‚38.56952395583333 disk.rbd1.in‚0 disk.dm-0.out‚ 55.958984375 disk.sda1.in‚0 network.recv‚53115 disk.sda2.in‚0 disk.rbd2.out‚ 0.8828125 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚54502 disk.sda.out‚ 7.99609375 disk.sda2.out‚0) proc.memory.availableMB‚ 3010.53515625 proc.memory.rssMB‚ 2760.96875) proc.memory.percent‚16.85161590576172 cpu‚25.767795227205877 disk.dm-1.in‚0 disk.sda3.in‚0$ disk./.usageGB‚574.7137794494629 disk.rbd1.out‚ 554.796875 disk.rbd0.in‚0 disk.dm-0.in‚ 159.55859375 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 159.55859375# disk.nvme0c0n1.out‚ 55.958984375 disk.sda.in‚0$ memory_percent‚81.62515163421631 disk.nvme0n1.in‚ 159.55859375 disk.rbd0.out‚0 disk.dm-1.out‚ 7.99609375! disk.nvme0n1.out‚ 55.958984375 disk.rbd2.in‚0 disk.sda3.out‚ 7.99609375;‘·‘!‚(:‡ ÍžêÒ¶€è« gpu.0.gpu‚90.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.133 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Dà ß"‚(:Õ ÍžêÒäøÓ® disk.sda.in‚0 cpu‚25.705074867987236 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2760.98828125 disk.sda1.out‚0 disk.nvme0n1.in‚ 288.3515625) proc.memory.percent‚16.85173511505127 disk.drbd2165.out‚0 disk.dm-1.out‚ 10.04296875 disk.sda3.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 3010.421875 network.sent‚60173 disk.dm-0.in‚ 288.3515625 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 288.3515625" disk.nvme0n1.out‚ 885.806640625 disk.sda.out‚ 10.04296875 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚57309) disk./.usagePercent‚38.62104214400125$ disk./.usageGB‚575.4814376831055 disk.dm-0.out‚ 885.806640625 disk.drbd2165.in‚0 disk.dm-1.in‚0# memory_percent‚81.6258430480957 disk.rbd1.out‚ 634.609375 disk.rbd0.out‚0 disk.rbd2.out‚ 0.8828125 disk.sda3.out‚ 10.04296875$ disk.nvme0c0n1.out‚ 885.806640625f"õå3#b/ lr-AdamW/pg1"trainer/global_step2:JPi¢3$b/ lr-AdamW/pg2"trainer/global_step2:JPa]»z-%) _wandbruntime‚85  _runtime‚85®4¤•À&» & lr-AdamW/pg1‚3.357313518106072e-05 & lr-AdamW/pg2‚3.357313518106072e-05 trainer/global_step‚49 % _timestamp‚1.7843198318953578e+09 _runtime‚ 85.270164417 _step‚0aT<¸Â'½ & lr-AdamW/pg1‚3.357313518106072e-05 & lr-AdamW/pg2‚3.357313518106072e-05 trainer/global_step‚49 % _timestamp‚1.7843198318953578e+09 _runtime‚ 85.270164417 _step‚0Ω—¿k(bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP^O['s)bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPTÁÝž]*bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP€÷Äk+bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPÒ5j,bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP°q Si-be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPå^¹Œn.bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP*òp/bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPòRŠ¨r0bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP«ÊÀYf1bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPÛlYØn2bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP…\ s3bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPþÁ%i4be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPi"¸¡h5bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPb¶9m6bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPüÉ>l7bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP÷w¸(a8b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPÞ·ÿe9ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP$ò_:b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP«9ú@p;bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPýò‰_bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPóum?bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP-œŒq@bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPùåperAbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP$Ûj®dBb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPìÈûàeCba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP¸J ðmDbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPè¿0vEbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPyßmrFbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPœnpGbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPéîjJ^HbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP©~´ágIbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPñógoJbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP‘7vmnKbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP Lý?gLbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPF2»KjMbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPX¡ý¶kNbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP¤ÑQAfObb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP:²î>pPbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPs’rQbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPÅVènRbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP=öXÆiSbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPÞé–çgTbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPýtìcUb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPØø‰^npbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP!݇Ódqb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP”¾ÄÇgrbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPêzXisbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP`îØpntbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP”\Ø`mubi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP‹÷[eqvbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP¬®4Whwbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPc8Åóoxbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP‘î²fybb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP\ÏÞ€ezba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP˲mj{bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPUFRIf|bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPºUÞb}b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPÛ[6êk~bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPÛâÖ«hbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP‚ ’®m€bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPN"© obj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP@ ¼d‚b_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPG`ñêi“bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP7%ÈJo”bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPUq9¤k•bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPõð‹p–bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPBû¬y—bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPo„²öa˜b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPœþÁ1o™bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPû0ŒøXšbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP„Ni×a›b\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP‡¡‚ãkœbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPW‹pbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPŸ•þBužbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¯¡'tŸbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP¿[/ n bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPÑ$¼g¡bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPÌ‘­bl¢bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPƒ5Sj.£) _wandbruntime‚85  _runtime‚85Ä"S½-¤O LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 45611199488 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚137091 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 45611199488 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 61559269376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚414671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚414665 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13986776473600 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 61524553216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 233325056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1713373184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚817 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679936000 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚269758 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49064968192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679936000 YIDeviceStatsMonitor.on_t…P yvrain_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1713373184 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚414665 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 46956409344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1280991744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1713373184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13986776473600 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4126860197376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544943616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚137351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469583 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13980577312768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚49 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚422323 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13986802647040 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚132407 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1992 $ _timestamp‚1.784319831896811e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544943616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚431945 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _runtime‚ 85.27465398 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4127263004672 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14026109206536 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45531893768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49064968192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14032387673088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚431945 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚422323 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801ŽÖ¨T§'¥¡O _runtime‚ 85.27465398 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4127263004672 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14026109206536 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45531893768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49064968192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14032387673088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚431945 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚422323 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 45611199488 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚137091 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 45611199488 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 61559269376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚414671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚414665 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13986776473600 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 61524553216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 233325056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1713373184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚817 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679936000 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚269758 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49064968192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679936000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1713373184 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚414665 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 46956409344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1280991744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1713373184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13986776473600 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4126860197376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544943616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚137351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469583 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13980577312768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚49 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚422323 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13986802647040 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚132407 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1992 $ _timestamp‚1.784319831896811e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544943616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚431945 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51¬gf9n¦bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP+ö„eh§bc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP¹Ì&¡f¨ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPgÞ¶Am©bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPµpªbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP匶]«bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPÌóð]e¬b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPÛVŠ^­bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPæùs½]®bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP~¡h¯bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP®§žIj°be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP‡‹o±bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPbµdm²bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÌì½d³b_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPâ%äeëb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPãÁKneìb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPÍ)3Lpíbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP’Riîbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPÕÛq9^ïbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPLsOúbðb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPª´Aßnñbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPÈÄëfòba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPîZÅŽdób_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPɯ‹Ûqÿbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPŒ ¥f€ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP(ïKQrbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP§¯â¤r‚bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP±&_‘gƒbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP€Æz›g„bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP¡tŠú^…bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPKGjBl†bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP¦Áf‡ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPõ^’iˆbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP¬7³Îg‰bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPçÐŒhŠbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPT¨æ6k‹bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP´†m.fŒba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP”+Ñ mbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP”ȲÊdŽb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPÆÆŸúi”bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP´ì¼j•be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP(žl–bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPu—{_—bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPªWÅDi˜bd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPœõør™bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPYôáÓgšbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP_ˆKÔr›bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPU„¦VœbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPŽÜû lbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPÞ8mžbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP SšrŸbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÆß׸m bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP$§ùû.¡) _wandbruntime‚86  _runtime‚86(˜4,…¢¨M REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 46455360014 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4205951550464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚817 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49064968192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1713373184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14306284477832 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 47799958920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49064968192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ÜjÊr) 1371390464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4143192240128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚422871 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚275070 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 62759310336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14264779300864 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46536235008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4142824148992 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 62724601344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚ 86.654906659 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚430801 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚422871 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1992 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚863314 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 46536235008 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚274810 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚134947 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14264805474304 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 47881452032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14304914450958 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 $ _timestamp‚1.784319833279997e+09 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚422877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14312686926336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1280991744 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚440437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4205548750336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14258484518912 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14311315535872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 237518336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚49 _step‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚440437 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640)Â_²&£¬M LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚440437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4205548750336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14258484518912 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14311315535872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 237518336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚49 _step‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚440437 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 46455360014 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4205951550464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚817 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49064968192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1713373184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14306284477832 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 47799958920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49064968192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4143192240128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚422871 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚275070 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 62759310336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14264779300864 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46536235008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4142824148992 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 62724601344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚ 86.654906659 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚430801 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚422871 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1992 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚863314 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 46536235008 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚274810 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚134947 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14264805474304 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 47881452032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14304914450958 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 $ _timestamp‚1.784319833279997e+09 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚422877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14312686926336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1280991744 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¾Ñ%ãC¤b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPäšP¥A¥b< loss/FBt1/assignment_loss"trainer/global_step2:JPHÆ\¸A¦b< loss/FBt2/assignment_loss"trainer/global_step2:JP’˜oB§b= loss/SRqqt1/detection_loss"trainer/global_step2:JP`Ý–ÍB¨b= loss/SRqqt2/detection_loss"trainer/global_step2:JPç¿àÀ@©b; loss/FBt1/detection_loss"trainer/global_step2:JP[…w_-ªb( epoch"trainer/global_step2:JP£Ýù¹A«b< loss/FRt1/assignment_loss"trainer/global_step2:JP…‚V@¬b; loss/FRt2/detection_loss"trainer/global_step2:JP¢eÅ@­b; loss/FBt2/detection_loss"trainer/global_step2:JPl‘Éé7®b2 loss/total_loss"trainer/global_step2:JPfMœ¿A¯b< loss/FRt2/assignment_loss"trainer/global_step2:JP§‹l C°b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP8\}6@±b; loss/FRt1/detection_loss"trainer/global_step2:JPÿzþý.²) _wandbruntime‚86  _runtime‚86l[èÀø³ò _step‚3 1loss/SRqqt1/assignment_loss‚4.227273757617096 /loss/FBt1/assignment_loss‚1.398169994354248 /loss/FBt2/assignment_loss‚1.516984430133787 1loss/SRqqt1/detection_loss‚1.2996468017241511 0loss/SRqqt2/detection_loss‚4.904123675364715 .loss/FBt1/detection_loss‚42.34347282117233 epoch‚0 trainer/global_step‚49 /loss/FRt1/assignment_loss‚5.344654705399282 0loss/FRt2/detection_loss‚0.41755288411382574 /loss/FBt2/detection_loss‚11.289988016155469 %loss/total_loss‚82.92491036903378 % _timestamp‚1.7843198333661406e+09 _runtime‚ 86.740469592 /loss/FRt2/assignment_loss‚4.541382696604278 1loss/SRqqt2/assignment_loss‚4.219152830459259 /loss/FRt1/detection_loss‚1.4225077559353447 ¬îü´ö /loss/FRt1/assignment_loss‚5.344654705399282 0loss/FRt2/detection_loss‚0.41755288411382574 /loss/FBt2/detection_loss‚11.289988016155469 %loss/total_loss‚82.92491036903378 % _timestamp‚1.7843198333661406e+09 _runtime‚ 86.740469592 /loss/FRt2/assignment_loss‚4.541382696604278 1loss/SRqqt2/assignment_loss‚4.219152830459259 /loss/FRt1/detection_loss‚1.4225077559353447 _step‚3 1loss/SRqqt1/assignment_loss‚4.227273757617096 /loss/FBt1/assignment_loss‚1.398169994354248 /loss/FBt2/assignment_loss‚1.516984430133787 1loss/SRqqt1/detection_loss‚1.2996468017241511 0loss/SRqqt2/detection_loss‚4.904123675364715 .loss/FBt1/detection_loss‚42.34347282117233 epoch‚0 trainer/global_step‚49b#&“µ‚(:ˆ ÜžêÒÆâø« gpu.0.gpu‚58.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚57.75398254394531* gpu.0.memoryAllocatedBytes‚ 49610293248 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D㾎(:ã ÜžêÒ®¡Î® disk.rbd1.out‚ 789.078125! disk.dm-0.out‚910.30419921875 disk.sda1.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 289.30859375& disk.nvme0c0n1.out‚910.30419921875) disk./.usagePercent‚38.61878031743623 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2761.03515625 network.recv‚61968$ disk./.usageGB‚575.4477348327637$ disk.nvme0n1.out‚910.30419921875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 13.05859375 disk.sda3.out‚ 13.05859375$ memory_percent‚81.62739276885986 disk.rbd2.out‚ 1.07421875) proc.memory.percent‚16.85202121734619 cpu‚25.660887955233765 proc.cpu.threads‚17) proc.memory.availableMB‚ 3010.16796875 network.sent‚107845 disk.rbd2.in‚0 disk.dm-0.in‚ 289.30859375 disk.sda.out‚ 13.05859375 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 289.30859375 disk.sda.in‚0ò#)U“·‚(:ˆ ëžêÒÊÄЬ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.75398254394531* gpu.0.memoryAllocatedBytes‚ 49610293248 gpu.0.temp‚43.0 gpu.0.powerWatts‚235.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó ˆžè¸‚(:Ý ëžêÒ¡²˜® disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚81.62992000579834 disk.rbd2.out‚ 1.07421875 disk.dm-0.in‚ 289.30859375 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2761.05078125$ disk./.usageGB‚574.7070732116699 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚38.56907389458814$ disk.nvme0n1.out‚915.06201171875 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 3009.75390625 disk.nvme0n1.in‚ 289.30859375 disk.rbd1.in‚0 disk.rbd1.out‚ 797.140625 cpu‚25.62899331593716 network.sent‚117287 disk.sda3.out‚ 14.921875" disk.nvme0c0n1.in‚ 289.30859375* proc.memory.percent‚16.852116584777832 network.recv‚67347 disk.sda.in‚0 disk.dm-1.out‚ 14.921875& disk.nvme0c0n1.out‚915.06201171875 proc.cpu.threads‚17 disk.sda.out‚ 14.921875! disk.dm-0.out‚915.062011718751øQ“¹‚(:ˆ úžêÒŽÛû« gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03MÄIܺ‚(:Ñ úžêÒûˆ÷® disk.drbd2165.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 290.140625 disk.sda2.out‚0 disk.rbd2.out‚ 1.13720703125 disk.nvme0c0n1.in‚ 290.140625( proc.memory.availableMB‚ 3009.6953125# proc.memory.rssMB‚ 2761.05859375 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚15.625 cpu‚25.599824382578124 proc.cpu.threads‚17 network.recv‚72008 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 290.140625 disk.dm-1.in‚0* proc.memory.percent‚16.852164268493652 disk.sda2.in‚0 disk.sda3.out‚15.625$ memory_percent‚81.63027763366699 network.sent‚124761$ disk./.usageGB‚575.3167304992676$ disk.nvme0n1.out‚927.14990234375 disk.rbd2.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚927.14990234375 disk.rbd0.out‚0) disk./.usagePercent‚38.60998850669535 disk.sda.out‚15.625 disk.rbd1.out‚ 1020.28125! disk.dm-0.out‚927.14990234375O}R“”»‚(:‰ ‰ŸêÒýýñ« gpu.0.gpu‚100.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô“N,꼂(:ß ‰ŸêÒÆç·® disk.sda3.in‚0 disk.sda.out‚16.09375 disk.rbd1.out‚ 1197.859375 disk.rbd0.in‚0 network.sent‚129047! disk.nvme0c0n1.in‚ 487.7734375 disk.nvme0n1.in‚ 487.7734375 proc.cpu.threads‚17 network.recv‚76091$ disk./.usageGB‚575.4245681762695 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2761.06640625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 1.47314453125 cpu‚25.581238779820193 disk.dm-1.out‚16.09375 disk.sda3.out‚16.09375 disk.rbd2.in‚0.015625 disk.dm-0.out‚962.0810546875 disk.dm-1.in‚0% disk.nvme0c0n1.out‚962.0810546875 disk.rbd0.out‚0* proc.memory.percent‚16.852211952209473 disk.sda1.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 487.7734375$ memory_percent‚81.29191398620605( proc.memory.availableMB‚ 3065.1328125) disk./.usagePercent‚38.61722558368079# disk.nvme0n1.out‚962.0810546875ÚE8‘½‚(:† ˜ŸêÒª¢ù« gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•»O徂(:Ú ˜ŸêÒ†îέ disk.rbd1.out‚ 1258.21875 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚81.29210472106934 cpu‚25.561648686777872 disk.sda1.in‚0 disk.nvme0n1.in‚ 507.7421875" disk.dm-0.out‚1732.42236328125 disk.sda1.out‚0 disk.dm-1.out‚16.875' disk.nvme0c0n1.out‚1732.42236328125! proc.memory.rssMB‚ 2761.078125 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 507.7421875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 1.591796875 network.sent‚134543) disk./.usagePercent‚38.56732024297859 disk.drbd2165.in‚0 disk.sda3.out‚16.875 disk.sda2.in‚0 network.recv‚79438 disk.drbd2165.out‚0 disk.sda.out‚16.875 disk.rbd1.in‚0( proc.memory.availableMB‚ 3065.1015625* proc.memory.percent‚16.852283477783203 proc.cpu.threads‚17$ disk./.usageGB‚574.6809425354004 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 507.7421875% disk.nvme0n1.out‚1732.42236328125 disk.sda.in‚0kíà“¿‚(:ˆ §ŸêÒÛ å« gpu.0.gpu‚38.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Xt$ðÀ‚(:å §ŸêÒ„’® disk.rbd2.in‚0.015625 network.recv‚83446! disk.dm-0.out‚1767.9970703125" disk.nvme0c0n1.in‚ 508.57421875& disk.nvme0c0n1.out‚1767.9970703125 disk.sda.out‚ 17.2890625) disk./.usagePercent‚38.56732049898613$ disk./.usageGB‚574.6809463500977 disk.drbd2165.in‚0 disk.dm-1.out‚ 17.2890625 disk.sda.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.852355003356934 disk.sda1.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.out‚ 1.591796875$ memory_percent‚81.29253387451172 disk.sda2.in‚0 disk.dm-0.in‚ 508.57421875 disk.sda1.in‚0 disk.nvme0n1.in‚ 508.57421875 cpu‚25.54609112819562 disk.sda3.in‚0 disk.sda3.out‚ 17.2890625 disk.rbd0.in‚0& proc.memory.availableMB‚ 3065.03125# proc.memory.rssMB‚ 2761.08984375 network.sent‚140051 disk.drbd2165.out‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚1767.9970703125 disk.rbd1.out‚ 1266.40625ÃÿÍ0Á+ _wandbruntime‚165  _runtime‚165º&x¼ % _timestamp‚1.7843199117122092e+09 _runtime‚ 165.086974783 _step‚4 & lr-AdamW/pg1‚6.783143638622473e-05 & lr-AdamW/pg2‚6.783143638622473e-05 trainer/global_step‚99¾ÏY%ÆÃÀ _step‚4 & lr-AdamW/pg1‚6.783143638622473e-05 & lr-AdamW/pg2‚6.783143638622473e-05 trainer/global_step‚99 % _timestamp‚1.7843199117122092e+09 _runtime‚ 165.086974783´ãÚ0Ä+ _wandbruntime‚165  _runtime‚165qãeË Å£O NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚824971 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27886917836800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚857697 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49067065344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1993 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27886917836800 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27966435228444 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1715470336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚818 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚824965 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 91927130112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7939760885760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 " _timestamp‚1.7843199117135e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 91927130112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27886944010240 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚824971 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27967805255206 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27980216350208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚546 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1715470336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7940128976896 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚5 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚848075 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_splihÞt.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚848075 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1682668 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 120861347840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93272339968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 266878464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1715470336 _runtime‚ 165.088256253 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27980216350208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 120898161152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚824965 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚536742 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 93113605670 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚259407 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚277339 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27874691649536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93272339968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚519 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1993 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27978844966912 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 91927130112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7939760885760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 " _timestamp‚1.7843199117135e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 91927130112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27886944010240 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚824971 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27967805255206 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27980216350208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚546 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1715470336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7940128976896 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚5 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚848075 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚848075 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1682668 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 120861347840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93272339968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 266878464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1715470336 _runtime‚ 165.088256253 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27980216350208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 120898161152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚824965 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚536742 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 93113605670 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚259407 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚277339 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27874691649536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93272339968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚519 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1993 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27978844966912 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚833171 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚833171 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 122074769920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1715470336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28152547999744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49067065344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚866189 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 270481920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚818 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚542076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚546 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28257772829696 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92852165632 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8139287354368 _step‚6 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚856553 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94197382656 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚261947 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚261943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚818 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28152573427712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚856553 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8017617481728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49067065344 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1715470336 _runtime‚ 166.474096283 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28246610499740 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚866189 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28164920664064 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8139692251648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 122037963776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28259144220160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28259144220160 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28245240472866 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28164920664064 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280133 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8017249390592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚818 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 92852165632 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94197382656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 92692473122 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49067065344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1993 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚833177 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28164946837504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚519 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 94037072028 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175úyP lÉ´M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28164920664064 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280133 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8017249390592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚818 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 92852165632 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94197382656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 92692473122 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49067065344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1993 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚833177 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28164946837504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚519 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 94037072028 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1715470336 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚833177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚833171 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚833171 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 122074769920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1715470336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28152547999744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49067065344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚866189 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 270481920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚818 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚542076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚546 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28257772829696 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92852165632 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8139287354368 _step‚6 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚856553 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94197382656 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.larïaÀSNge_pool.allocated‚261947 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚261943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚818 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28152573427712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚856553 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8017617481728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49067065344 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1715470336 _runtime‚ 166.474096283 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28246610499740 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚866189 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28164920664064 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8139692251648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 122037963776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28259144220160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28259144220160 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28245240472866^±0Ê+ _wandbruntime‚166  _runtime‚166_«ÖÛûËõ 1loss/FBt2/assignment_loss‚0.18597802964852717 0loss/SRqqt2/detection_loss‚4.553361671266856 $ _timestamp‚1.784319913185063e+09 /loss/FRt2/assignment_loss‚4.484637445449829 2loss/SRqqt1/assignment_loss‚2.6369039745951306 .loss/FRt2/detection_loss‚0.334731425334195 1loss/SRqqt1/detection_loss‚1.0940233725217787 .loss/FBt2/detection_loss‚7.112489735094691 epoch‚0 1loss/SRqqt2/assignment_loss‚2.544244559911581 /loss/FRt1/detection_loss‚1.5919489398141646 /loss/FBt1/detection_loss‚16.579856226003017 _runtime‚ 166.559413177 _step‚7 /loss/FRt1/assignment_loss‚5.347023908830063 %loss/total_loss‚46.76947348512773 trainer/global_step‚99 1loss/FBt1/assignment_loss‚0.30427419665789646Ù"ûzÿÌù /loss/FRt1/assignment_loss‚5.347023908830063 %loss/total_loss‚46.76947348512773 trainer/global_step‚99 1loss/FBt1/assignment_loss‚0.30427419665789646 1loss/FBt2/assignment_loss‚0.18597802964852717 0loss/SRqqt2/detection_loss‚4.553361671266856 $ _timestamp‚1.784319913185063e+09 /loss/FRt2/assignment_loss‚4.484637445449829 2loss/SRqqt1/assignment_loss‚2.6369039745951306 .loss/FRt2/detection_loss‚0.334731425334195 1loss/SRqqt1/detection_loss‚1.0940233725217787 .loss/FBt2/detection_loss‚7.112489735094691 epoch‚0 1loss/SRqqt2/assignment_loss‚2.544244559911581 /loss/FRt1/detection_loss‚1.5919489398141646 /loss/FBt1/detection_loss‚16.579856226003017 _runtime‚ 166.559413177 _step‚7«~—²”Í‚(:‰ ¶ŸêÒá÷‡¬ gpu.0.gpu‚100.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O ø΂(:í ¶ŸêÒ‚çû°" disk.nvme0c0n1.in‚ 508.57421875$ disk.nvme0n1.out‚1772.7353515625 disk.sda.out‚ 17.69140625 disk.sda2.out‚0* proc.memory.percent‚16.852355003356934 cpu‚25.53465726329325 disk.dm-1.out‚ 17.69140625 disk.sda2.in‚0) disk./.usagePercent‚38.56552844624645 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd1.out‚ 1489.78125 disk.rbd0.in‚0 network.recv‚87563! disk.dm-0.out‚1772.7353515625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 508.57421875) proc.memory.availableMB‚ 3064.22265625 disk.sda3.out‚ 17.69140625& disk.nvme0c0n1.out‚1772.7353515625 network.sent‚145795 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚81.29746913909912# proc.memory.rssMB‚ 2761.08984375 disk.rbd2.out‚ 1.63525390625$ disk./.usageGB‚574.6542434692383 disk.dm-0.in‚ 508.57421875 disk.drbd2165.in‚0Lè&“Ï‚(:ˆ ÅŸêÒÿ潫 gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#åü#ïЂ(:ä ÅŸêÒ—»«® disk.rbd0.out‚0 disk.rbd2.out‚ 1.63525390625! disk.dm-0.out‚1786.1494140625 disk.sda1.in‚0 cpu‚25.522694474127167$ disk./.usageGB‚574.6542091369629 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 17.92578125 disk.sda.in‚0 disk.sda.out‚ 17.92578125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 509.53125 proc.cpu.threads‚17 network.recv‚91784 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚1786.1494140625 disk.sda2.out‚0* proc.memory.percent‚16.852402687072754 disk.nvme0n1.in‚ 509.53125$ disk.nvme0n1.out‚1786.1494140625( proc.memory.availableMB‚ 3064.3046875# proc.memory.rssMB‚ 2761.09765625 disk.rbd1.out‚ 1690.28125) disk./.usagePercent‚38.56552614217863 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 509.53125 disk.sda1.out‚0 disk.sda3.out‚ 17.92578125$ memory_percent‚81.29696846008301 network.sent‚192497êÌí”Ñ‚(:‰ ÔŸêÒÓº¬ gpu.0.gpu‚100.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#LµïÒ‚(:ä ÔŸêÒ¹ôî® cpu‚25.513672029053968 network.recv‚95485 disk.dm-0.in‚ 685.2265625 disk.sda1.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚1795.623046875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 18.2421875 disk.nvme0n1.in‚ 685.2265625 disk.sda2.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚38.565527678223845 disk.rbd2.in‚0.015625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 1768.265625 disk.sda.out‚ 18.2421875 network.sent‚198020 disk.sda1.out‚0) proc.memory.availableMB‚ 3066.19140625* proc.memory.percent‚16.852426528930664$ disk./.usageGB‚574.6542320251465 disk.rbd2.out‚ 1.6982421875 disk.dm-0.out‚1795.623046875 disk.drbd2165.in‚0 disk.sda3.out‚ 18.2421875# memory_percent‚81.2854528427124" proc.memory.rssMB‚ 2761.1015625! disk.nvme0c0n1.in‚ 685.2265625% disk.nvme0c0n1.out‚1795.623046875Ìa­“Ó‚(:ˆ ãŸêÒ§¤¥¬ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sqm®óÔ‚(:è ãŸêÒ¹ò¥®# disk.nvme0n1.out‚1803.111328125 disk.dm-0.in‚ 685.2265625 disk.drbd2165.in‚0$ memory_percent‚81.28588199615479 cpu‚25.503966187980527 disk.dm-1.in‚0% disk.nvme0c0n1.out‚1803.111328125 disk.nvme0n1.in‚ 685.2265625 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 685.2265625 disk.rbd2.out‚ 1.6982421875 disk.sda1.out‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 3066.12109375# disk./.usageGB‚574.654468536377 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 18.37890625* proc.memory.percent‚16.852450370788574 disk.sda3.in‚0 disk.rbd1.out‚ 1776.453125 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.out‚1803.111328125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 18.37890625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 18.37890625# proc.memory.rssMB‚ 2761.10546875 network.sent‚203771 network.recv‚100022) disk./.usagePercent‚38.56554355069097 disk.rbd0.in‚0-Z/ΓÕ‚(:ˆ òŸêÒš¼Œ¬ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚46.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05˜u(òÖ‚(:ç òŸêÒ³°¿® network.recv‚103482 disk.sda3.out‚ 18.6796875" disk.nvme0c0n1.in‚ 686.18359375$ disk.nvme0c0n1.out‚ 1811.41796875 disk.sda.out‚ 18.6796875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2761.10546875 cpu‚25.498036800376617 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 686.18359375 disk.rbd1.in‚0 disk.dm-0.in‚ 686.18359375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 18.6796875 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚209215 disk.sda3.in‚0 disk.rbd1.out‚ 1954.359375 disk.rbd2.out‚ 1.76123046875 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0) proc.memory.availableMB‚ 3066.33203125$ disk./.usageGB‚574.6544876098633 disk.dm-0.out‚ 1811.41796875 disk.sda.in‚0* proc.memory.percent‚16.852450370788574) disk./.usagePercent‚38.56554483072864" disk.nvme0n1.out‚ 1811.41796875 disk.sda2.in‚0 disk.rbd0.out‚0$ memory_percent‚81.28459453582764tÏøa0×+ _wandbruntime‚244  _runtime‚244½ÂÃؽ ' lr-AdamW/pg2‚0.00010208973759138873 trainer/global_step‚149 # _timestamp‚1.78431999117546e+09 _runtime‚ 244.550077319 _step‚8 ' lr-AdamW/pg1‚0.00010208973759138873ŒPÇÙÁ ' lr-AdamW/pg2‚0.00010208973759138873 trainer/global_step‚149 # _timestamp‚1.78431999117546e+09 _runtime‚ 244.550077319 _step‚8 ' lr-AdamW/pg1‚0.00010208973759138873n¸>‹0Ú+ _wandbruntime‚244  _runtime‚244D ÃI¿'Û¹O K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚802793 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1717567488 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚819 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1283449 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139588270592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1994 % _timestamp‚1.7843199911767304e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 139588270592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚819 _runtime‚ 244.551484126 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11814554218496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41908131277114 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1235265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284096512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41926673644032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1235265 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38910464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41906761250352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚386407 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚819 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1235271 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41925302260736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚416390 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1717567488 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2509092 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1273827 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1329225216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1994 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11994142096384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 138243060736 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 138243060736 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1717567488 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1283449 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚558 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41925302260736 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚386403 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚416679 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41768755130368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49069162496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41787085373440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11814186127360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 179548967424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41787085373440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚803086 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1235265 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38910464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41906761250352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚386407 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚819 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1235271 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41925302260736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚416390 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1717567488 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2509092 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1273827 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1329225216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1994 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11994142096384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 138243060736 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 138243060736 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1717567488 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1283449 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚558 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41925302260736 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚386403 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚416679 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41768755130368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49069162496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41787085373440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11814186127360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 179548967424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41787085373440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚803086 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚819 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1283449 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139588270592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1994 % _timestamp‚1.7843199911767304e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 139588270592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚819 _runtime‚ 244.551484126 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11814554218496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41908131277114 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1235265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284096512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41926673644032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGxb" DeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 HÅî0Ý+ _wandbruntime‚245  _runtime‚245×vµ„Ê&ÞÄM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 180765077504 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1282305 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140513313280 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42065088200704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚388943 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 139168096256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚388947 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚149 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42046636908544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42205601513984 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1291941 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚558 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406994432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1717567488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42205601513984 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139168096256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚819 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚808085 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49069162496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1717567488 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1994 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1994 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚819 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42065062027264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49069162496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚819 _step‚10 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1291941 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140513313280 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 180726174208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1282305 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚ 245.940292996 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11891674632192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1717567488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1243471 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 180726174208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1282305 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚ 245.940292996 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11891674632192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1717567488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1243471 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚149 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42046636908544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42205601513984 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1291941 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚558 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406994432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1717567488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42205601513984 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139168096256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚819 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚808085 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49069162496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1717567488 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1994 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1994 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚819 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42065062027264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49069162496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚819 _step‚10 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1291941 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140513313280 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 BèÚæ‚(:Ï  êÒ¯í® disk.sda2.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚574.6546592712402 disk.dm-1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 19.5703125 disk.nvme0n1.in‚822 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 2295.96875 disk.dm-0.in‚822) disk./.usagePercent‚38.56555635106768 disk.sda3.in‚0 disk.sda3.out‚ 19.5703125 disk.sda.out‚ 19.5703125 disk.rbd1.in‚0 disk.rbd2.in‚0.015625$ memory_percent‚81.29069805145264! proc.memory.rssMB‚ 2761.109375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0! disk.dm-0.out‚1823.9599609375* proc.memory.percent‚16.852474212646484 network.sent‚261945 disk.nvme0c0n1.in‚822 disk.sda2.in‚0) proc.memory.availableMB‚ 3065.33203125 proc.cpu.threads‚17 network.recv‚112384& disk.nvme0c0n1.out‚1823.9599609375$ disk.nvme0n1.out‚1823.9599609375 disk.rbd2.out‚ 1.76123046875 cpu‚25.483040083852455²érç‚(:’ Ÿ êÒ˜¼œ® gpu.0.gpu‚29.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`°ûè‚(:ð Ÿ êÒ¦–í® disk.dm-1.out‚ 20.24609375* proc.memory.percent‚16.852498054504395 disk.rbd1.out‚ 2367.28125# memory_percent‚81.2910795211792 cpu‚25.47889572779609 disk.sda3.in‚0" disk.dm-0.out‚1833.32470703125) proc.memory.availableMB‚ 3065.26953125 disk.nvme0n1.in‚ 822.95703125 disk.rbd2.out‚ 1.76123046875 network.recv‚116402 disk.sda.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚574.6546211242676 disk.sda3.out‚ 20.24609375 disk.sda.out‚ 20.24609375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚267981% disk.nvme0n1.out‚1833.32470703125 disk.drbd2165.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1833.32470703125 disk.sda2.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2761.11328125) disk./.usagePercent‚38.56555379099233" disk.nvme0c0n1.in‚ 822.95703125 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 822.95703125¼ÃI”é‚(:‰ ® êÒµù£¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡~nõê‚(:ê ® êұ阯 network.recv‚120043 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 cpu‚25.47288933680396 disk.sda2.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.852498054504395 network.sent‚272813 disk.sda.in‚0 disk.sda.out‚ 20.56640625$ memory_percent‚81.29110336303711' proc.memory.availableMB‚ 3065.265625 disk.rbd1.out‚ 2508.203125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd2.out‚ 1.76123046875# proc.memory.rssMB‚ 2761.11328125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 822.95703125% disk.nvme0c0n1.out‚1839.771484375 disk.nvme0n1.in‚ 822.95703125 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.56557324756494 disk.dm-0.in‚ 822.95703125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚574.6549110412598 disk.dm-0.out‚1839.771484375 disk.dm-1.out‚ 20.56640625 disk.sda3.out‚ 20.56640625# disk.nvme0n1.out‚1839.771484375BÙÏSŸë‚(:” ½ êÒïÙ¶® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚304.576' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.915200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´²î_öì‚(:ë ½ êÒð² disk.sda3.out‚ 20.8671875" disk.dm-0.out‚1847.40185546875 disk.sda1.out‚0 network.recv‚123928 disk.drbd2165.in‚0% disk.nvme0n1.out‚1847.40185546875 disk.rbd0.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚38.565597824288226 disk.sda.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 1.76123046875 disk.sda1.in‚0" proc.memory.rssMB‚ 2761.1171875 network.sent‚278259 disk.dm-1.out‚ 20.8671875 disk.rbd1.in‚0 disk.rbd1.out‚ 2596.6875$ disk./.usageGB‚574.6552772521973 disk.sda.out‚ 20.8671875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚1847.40185546875 disk.dm-0.in‚ 823.7890625 cpu‚25.469170448925762! disk.nvme0c0n1.in‚ 823.7890625 disk.sda2.out‚0* proc.memory.percent‚16.852521896362305 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 823.7890625$ memory_percent‚81.28197193145752) proc.memory.availableMB‚ 3066.76171875 disk.sda3.in‚0Ѧ/a0í+ _wandbruntime‚324  _runtime‚324“hZiÅî¿ _runtime‚ 324.593573885 _step‚12 ' lr-AdamW/pg1‚0.00013634803879655272 ' lr-AdamW/pg2‚0.00013634803879655272 trainer/global_step‚199 $ _timestamp‚1.784320071218896e+09^G¹œÉïà ' lr-AdamW/pg1‚0.00013634803879655272 ' lr-AdamW/pg2‚0.00013634803879655272 trainer/global_step‚199 $ _timestamp‚1.784320071218896e+09 _runtime‚ 324.593573885 _step‚12 Þx̪0ð+ _wandbruntime‚324  _runtime‚324Žfîè¼ ñ¼O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1699579 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185587831886 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1709201 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1699579 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1068156 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55662844039168 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1645571 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15926880700416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚ 324.594840575 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55873130937856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚41007616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15926471601664 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 237860232704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 184243233092 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55687200563200 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 184558991360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1645565 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 237901240320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55873130937856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55662869467136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1645571 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚513407 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1645565 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55687226736640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1709201 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _step‚13 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55871759554560 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1067850 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6žÆ'òÀO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1699579 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185587831886 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1709201 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1699579 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1068156 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55662844039168 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1645571 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15926880700416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚ 324.594840575 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55873130937856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚41007616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15926471601664 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 237860232704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 184243233092 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55687200563200 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 184558991360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1645565 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 237901240320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55873130937856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55662869467136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1645571 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚513407 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1645565 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55687226736640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1709201 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _step‚13 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55871759554560 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1067850 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ¿Õ¢±0ó+ _wandbruntime‚325  _runtime‚325Þ7ú›Ì&ôÆM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56152058807808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55940725817344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1717693 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 186829243904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56127262543556 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55965229563904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 185484026880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1653777 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1073414 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56152058807808 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 185484026880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56125892516682 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3371470 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚199 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 186511298244 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1073116 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55965203390464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15766467964928 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15766099873792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55965229563904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6‹DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56152058807808 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 185484026880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56125892516682 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3371470 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚199 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 186511298244 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1073116 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55965203390464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15766467964928 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15766099873792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55965229563904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56152058807808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55940725817344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1717693 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 186829243904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56127262543556 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55965229563904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 185484026880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1653777 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1073414 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968"œ0ö+ _wandbruntime‚326  _runtime‚326Ý *þ÷ø /loss/FRt2/detection_loss‚0.3857593971010639 /loss/FRt2/assignment_loss‚4.561281492451108 2loss/SRqqt2/assignment_loss‚1.6879237135615917 1loss/SRqqt2/detection_loss‚3.7070772168012933 .loss/FBt2/detection_loss‚8.730287384278505 %loss/total_loss‚35.79245926289933 trainer/global_step‚199 0loss/FBt2/assignment_loss‚0.3806093543544641 1loss/SRqqt1/detection_loss‚1.0514405342380029 .loss/FBt1/detection_loss‚6.172111383422997 epoch‚0 _runtime‚ 326.070359985 _step‚15 .loss/FRt1/assignment_loss‚5.31165221832894 2loss/SRqqt1/assignment_loss‚2.1856337214733386 1loss/FBt1/assignment_loss‚0.17713596426951017 /loss/FRt1/detection_loss‚1.4415468826185207 % _timestamp‚1.7843200726959288e+09ÓÙôøü 2loss/SRqqt2/assignment_loss‚1.6879237135615917 1loss/SRqqt2/detection_loss‚3.7070772168012933 .loss/FBt2/detection_loss‚8.730287384278505 %loss/total_loss‚35.79245926289933 trainer/global_step‚199 0loss/FBt2/assignment_loss‚0.3806093543544641 1loss/SRqqt1/detection_loss‚1.0514405342380029 .loss/FBt1/detection_loss‚6.172111383422997 epoch‚0 _runtime‚ 326.070359985 _step‚15 .loss/FRt1/assignment_loss‚5.31165221832894 2loss/SRqqt1/assignment_loss‚2.1856337214733386 1loss/FBt1/assignment_loss‚0.17713596426951017 /loss/FRt1/detection_loss‚1.4415468826185207 % _timestamp‚1.7843200726959288e+09 /loss/FRt2/detection_loss‚0.3857593971010639 /loss/FRt2/assignment_loss‚4.561281492451108?3òEžù‚(:“ Ì êÒð‰±« gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Øþùú‚(:î Ì êÒŸÚÆ­ disk.sda.out‚ 21.05859375 disk.rbd1.in‚0) disk./.usagePercent‚38.56553843054028 disk.dm-1.in‚0* proc.memory.percent‚16.852521896362305 disk.sda.in‚0" disk.dm-0.out‚1855.45947265625 disk.sda1.in‚0 disk.sda3.out‚ 21.05859375" proc.memory.rssMB‚ 2761.1171875 network.sent‚283649$ disk./.usageGB‚574.6543922424316 disk.rbd0.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 917.94140625$ memory_percent‚81.28113746643066( proc.memory.availableMB‚ 3066.8984375 disk.drbd2165.out‚0 cpu‚25.46485726835618 network.recv‚127425 disk.rbd2.in‚0.015625% disk.nvme0n1.out‚1855.45947265625 disk.sda2.out‚0 disk.rbd1.out‚ 2679.6875 disk.sda2.in‚0 disk.rbd2.out‚ 1.76123046875 disk.dm-0.in‚ 917.94140625 disk.dm-1.out‚ 21.05859375 disk.sda3.in‚0 disk.nvme0n1.in‚ 917.94140625 proc.cpu.threads‚17 disk.rbd0.in‚0' disk.nvme0c0n1.out‚1855.45947265625 disk.drbd2165.in‚0¼ÎÔžû‚(:“ Û êÒ”Ûõ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôqŸøü‚(:í Û êÒÜ´À®$ disk./.usageGB‚574.6544990539551 proc.cpu.threads‚17 disk.sda1.in‚0' disk.nvme0c0n1.out‚1861.47998046875% disk.nvme0n1.out‚1861.47998046875$ memory_percent‚81.27987384796143 disk.rbd2.in‚0.015625" disk.dm-0.out‚1861.47998046875 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 21.2265625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 917.94140625 disk.sda2.out‚0 network.sent‚331857 network.recv‚132299 disk.sda3.out‚ 21.2265625" disk.nvme0c0n1.in‚ 917.94140625) proc.memory.availableMB‚ 3067.10546875* proc.memory.percent‚16.852521896362305 cpu‚25.460061613357* disk./.usagePercent‚38.565545598751235 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 21.2265625 disk.sda2.in‚0 disk.dm-0.in‚ 917.94140625 disk.rbd1.in‚0 disk.rbd1.out‚ 2877.671875" proc.memory.rssMB‚ 2761.1171875 disk.rbd0.out‚0 disk.rbd2.out‚ 1.76123046875Çu2,ïý‚(:ä ê êÒªãó­ disk.dm-0.out‚ 1869.08984375 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 2993.703125 disk.rbd0.out‚0 cpu‚25.457155224765103 network.recv‚136029 disk.sda1.in‚0$ memory_percent‚81.28001689910889) proc.memory.availableMB‚ 3067.08203125$ disk.nvme0c0n1.out‚ 1869.08984375 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.rbd0.in‚0* proc.memory.percent‚16.852545738220215) disk./.usagePercent‚38.56553868654782 disk.nvme0n1.in‚ 918.7734375$ disk./.usageGB‚574.6543960571289 disk.sda3.out‚ 21.6171875" disk.nvme0n1.out‚ 1869.08984375 disk.sda.out‚ 21.6171875 proc.cpu.threads‚17 disk.rbd2.out‚ 1.76123046875 disk.dm-1.out‚ 21.6171875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 918.7734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 918.7734375 disk.sda.in‚0# proc.memory.rssMB‚ 2761.12109375 network.sent‚336864ü$dÔþ‚(:‰ ê êÒÖÓó­ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤õ!™”ÿ‚(:‰ ù êÒ€ƒ°¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0꬙ö€‚(:ë ù êÒÑή disk.dm-1.out‚ 22.203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 919.73046875" disk.dm-0.out‚1879.34228515625 disk.rbd1.out‚ 3134.265625 disk.drbd2165.in‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚1879.34228515625* proc.memory.percent‚16.852593421936035 cpu‚25.45371549152651 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.recv‚139481( disk./.usagePercent‚38.5655430386759 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 919.73046875# proc.memory.rssMB‚ 2761.12890625 network.sent‚342029 disk.rbd2.out‚ 1.76123046875$ disk./.usageGB‚574.6544609069824% disk.nvme0n1.out‚1879.34228515625$ memory_percent‚81.28206729888916 disk.dm-1.in‚0 disk.sda3.out‚ 22.203125) proc.memory.availableMB‚ 3066.74609375 disk.nvme0n1.in‚ 919.73046875 disk.sda.out‚ 22.203125 disk.rbd1.in‚04[@ý‚(:ò ˆ¡êÒ–‹À­ disk.sda.out‚ 23.16796875 disk.rbd1.out‚ 3208.453125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2761.12890625 proc.cpu.threads‚17 disk.sda3.out‚ 23.16796875 disk.rbd0.out‚0 disk.rbd2.out‚ 1.82373046875 disk.drbd2165.out‚0( proc.memory.availableMB‚ 3067.2265625* proc.memory.percent‚16.852593421936035 network.recv‚143031$ disk.nvme0n1.out‚1884.2724609375 disk.rbd1.in‚0 disk.rbd2.in‚0.015625$ memory_percent‚81.27913475036621 network.sent‚346810 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.451265443799006* disk./.usagePercent‚38.565495421274534 disk.sda2.in‚0 disk.sda1.out‚0# disk./.usageGB‚574.653751373291# disk.nvme0c0n1.in‚ 1017.22265625& disk.nvme0c0n1.out‚1884.2724609375! disk.dm-0.out‚1884.2724609375 disk.dm-1.out‚ 23.16796875 disk.sda3.in‚0! disk.nvme0n1.in‚ 1017.22265625 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1017.22265625`I ••‚‚(:Š ˆ¡êҜЛ® gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²;|É0ƒ+ _wandbruntime‚403  _runtime‚403¤ Q­Æ„À % _timestamp‚1.7843201505354018e+09 _runtime‚ 403.910060284 _step‚16 ' lr-AdamW/pg1‚0.00017060634000171676 ' lr-AdamW/pg2‚0.00017060634000171676 trainer/global_step‚249+÷Ò»Ê…Ä trainer/global_step‚249 % _timestamp‚1.7843201505354018e+09 _runtime‚ 403.910060284 _step‚16 ' lr-AdamW/pg1‚0.00017060634000171676 ' lr-AdamW/pg2‚0.00017060634000171676Þ0†+ _wandbruntime‚403  _runtime‚403 ñ²®Á'‡»O Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69818216848384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69556958375936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69587368099840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69587341926400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4190824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69788783320930 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69819588231680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19563404701696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 232220131840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69556932947968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 trainer/global_step‚249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69587368099840 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 230874921984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19563036610560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚17 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1332733 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2055865 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2125331 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 403.914365562 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19859826611712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 296385096704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 232220131840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69587341926400 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2055871 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 230874921984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2055871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1333003 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640Bvp !ˆ¿O RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚640403 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2055871 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 230874921984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2055871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1333003 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69818216848384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69556958375936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69587368099840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69587341926400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4190824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69788783320930 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69819588231680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19563404701696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 232220131840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69556932947968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 trainer/global_step‚249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69587368099840 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 230874921984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19563036610560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚17 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1332733 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2055865 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2125331 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 403.914365562 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 E;DeviceStatsMonú¡ßù¹itor.on_train_batch_start/allocation.all.peak‚11803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19859826611712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 296385096704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 232220131840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69587341926400 H6”‰‚(:‰ —¡êÒ̲¸« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z‘ zõŠ‚(:ê —¡êÒìµÈ­" disk.nvme0c0n1.in‚ 1018.0546875% disk.nvme0c0n1.out‚1897.556640625 disk.sda.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 1018.0546875) disk./.usagePercent‚38.56549772534233 disk.sda3.out‚ 23.4296875 disk.rbd1.in‚0 disk.rbd1.out‚ 3383.953125 disk.dm-0.out‚1897.556640625 disk.dm-1.out‚ 23.4296875 network.sent‚352196 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1018.0546875 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚81.28077983856201 disk.sda.out‚ 23.4296875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 3066.95703125* proc.memory.percent‚16.852593421936035 cpu‚25.447722182401346 network.recv‚146832# disk.nvme0n1.out‚1897.556640625 disk.sda2.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚574.6537857055664 disk.rbd2.out‚ 1.82373046875 proc.cpu.threads‚17 disk.rbd0.out‚0# proc.memory.rssMB‚ 2761.12890625>Âé@0‹+ _wandbruntime‚405  _runtime‚405‰¡aéÌ&ŒÆM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19938088199168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843201519239705e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231403812446 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69834814726144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69865370927104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2143445 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2064071 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 297599889920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69834840154112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1338026 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69865370927104 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 233145174528 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2133809 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70097144711168 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 233145174528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 297563083776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚642947 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70098516101632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 232748411352 trainer/global_step‚249 _step‚18 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2133809 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 231799957504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 231799957504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69865344753664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19640893206528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70067588565464 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2064077 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2143445 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2064071 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19938088199168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843201519239705e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231403812446 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69834814726144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69865370927104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2143445 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2064071 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 297599889920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69834840154112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1338026 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69865370927104 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 233145174528 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2133809 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70097144711168 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 233145174528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 297563083776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚642947 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70098516101632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 232748411352 trainer/global_step‚249 _step‚18 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2133809 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 231799957504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 231799957504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69865344753664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19640893206528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70067588565464 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2064077 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2143445 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2064071 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83764674142208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83487509463040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23437461852160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2466165 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83764674142208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2551083 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278536062464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2466171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83766045525504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23793247851520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 278062058102 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚21 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83766045525504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2466171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83727739316076 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 355820715520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398Àt(HÆ' ÀO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83766045525504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2466171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83727739316076 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 355820715520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23437829943296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83451021856768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83487483289600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83487483289600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1598048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83487509463040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83451047284736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278536062464 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 277190852608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83729109342838 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1598309 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2466165 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚767407 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23793650658816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843202313142123e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2560705 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚ 484.692429278 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚299 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83764674142208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83487509463040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23437461852160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2466165 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83764674142208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2551083 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278536062464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2466171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83766045525504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23793247851520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 278062058102 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚21ÝZ'B0¡+ _wandbruntime‚486  _runtime‚486nI&ÍÊ&¢ÄM F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84044973395456 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2474371 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83765486116864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2474377 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84006544560498 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚833404 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2569197 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279461105152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚299 _step‚22 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83728903634944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2559561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84007914587372 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 278115888128 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2559561 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83728929062912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83765512290304 # _timestamp‚1.78432023270795e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84044973395456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83765512290304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 277640925554 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23872325777920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2569197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1603347 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23515318448128 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 357007329792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278985524460 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 486.082835307 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83765486116864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23514950356992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23871922977792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 356972620800 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84043602004992 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚769943 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279461105152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2474371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16035988¿á¯ £ÈM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84044973395456 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2474371 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83765486116864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2474377 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84006544560498 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚833404 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2569197 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279461105152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚299 _step‚22 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83728903634944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2559561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytŽ±ú¸es.small_pool.peak‚ 1679937024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84007914587372 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 278115888128 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2559561 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83728929062912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83765512290304 # _timestamp‚1.78432023270795e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84044973395456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83765512290304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 277640925554 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23872325777920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2569197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1603347 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23515318448128 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 357007329792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278985524460 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 486.082835307 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83765486116864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23514950356992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23871922977792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 356972620800 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84043602004992 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚769943 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279461105152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2474371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1603598 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574mzËU0¤+ _wandbruntime‚486  _runtime‚486õpa<¥û &loss/total_loss‚57.529837759580396 epoch‚0 % _timestamp‚1.7843202327889657e+09 0loss/FRt1/assignment_loss‚4.6101243902997275 1loss/SRqqt1/assignment_loss‚1.794793874576805 2loss/SRqqt2/assignment_loss‚1.6796232984871955 /loss/FRt1/detection_loss‚1.3609064007632252 /loss/FRt2/detection_loss‚0.3621900288088246 0loss/SRqqt2/detection_loss‚2.871172355745809 _runtime‚ 486.163666044 /loss/FRt2/assignment_loss‚4.534755928247045 .loss/FBt1/detection_loss‚28.42605306575509 /loss/FBt2/detection_loss‚10.193469986879736 trainer/global_step‚299 _step‚23 1loss/FBt1/assignment_loss‚0.21941439822297606 1loss/FBt2/assignment_loss‚0.27666383524096216 1loss/SRqqt1/detection_loss‚1.2006701965529996ÇÅDï¦ÿ 1loss/SRqqt1/detection_loss‚1.2006701965529996 &loss/total_loss‚57.529837759580396 epoch‚0 % _timestamp‚1.7843202327889657e+09 0loss/FRt1/assignment_loss‚4.6101243902997275 1loss/SRqqt1/assignment_loss‚1.794793874576805 2loss/SRqqt2/assignment_loss‚1.6796232984871955 /loss/FRt1/detection_loss‚1.3609064007632252 /loss/FRt2/detection_loss‚0.3621900288088246 0loss/SRqqt2/detection_loss‚2.871172355745809 _runtime‚ 486.163666044 /loss/FRt2/assignment_loss‚4.534755928247045 .loss/FBt1/detection_loss‚28.42605306575509 /loss/FBt2/detection_loss‚10.193469986879736 trainer/global_step‚299 _step‚23 1loss/FBt1/assignment_loss‚0.21941439822297606 1loss/FBt2/assignment_loss‚0.27666383524096216¡Z”§‚(:‰ ñ¡êÒŽÚ¬ gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ˆÊÜú¨‚(:ï ñ¡êÒ–Êœ® disk.dm-1.out‚ 27.70703125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0$ memory_percent‚81.28485679626465 proc.cpu.threads‚17 network.recv‚170287 disk.dm-0.in‚ 1112.01953125 disk.dm-1.in‚0* proc.memory.percent‚16.852593421936035 disk.rbd0.in‚0 disk.sda.in‚0 cpu‚25.435063925315333 disk.sda1.out‚0$ disk./.usageGB‚574.8676528930664! disk.nvme0n1.in‚ 1112.01953125 disk.rbd1.out‚ 4198.0625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1112.01953125 disk.sda.out‚ 27.70703125! disk.dm-0.out‚1984.6787109375 disk.sda3.out‚ 27.70703125& disk.nvme0c0n1.out‚1984.6787109375$ disk.nvme0n1.out‚1984.6787109375( proc.memory.availableMB‚ 3066.2890625# proc.memory.rssMB‚ 2761.12890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚447732) disk./.usagePercent‚38.57985053174198 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 1.9736328125g“µªž©‚(:“ €¢êÒî¾³® gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«“Óeóª‚(:è €¢êÒ·ñ ± disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚38.58763674488837 disk.sda1.out‚0 disk.sda3.out‚ 28.1171875 disk.sda.out‚ 28.1171875 cpu‚25.432845145843714 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 28.1171875 network.sent‚496029 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 1225.83984375# proc.memory.rssMB‚ 2761.12890625* proc.memory.percent‚16.852593421936035 disk.rbd2.out‚ 1.9736328125 disk.drbd2165.in‚0' proc.memory.availableMB‚ 3066.328125 disk.dm-1.in‚0$ disk./.usageGB‚574.9836730957031 disk.dm-0.in‚ 1225.83984375% disk.nvme0c0n1.out‚2149.994140625 disk.rbd1.out‚ 4203.4375 disk.dm-0.out‚2149.994140625 disk.sda.in‚0$ memory_percent‚81.28461837768555 network.recv‚174737# disk.nvme0c0n1.in‚ 1225.83984375# disk.nvme0n1.out‚2149.994140625zs-ŒŸ«‚(:” ¢êÒï´ì« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚178.328' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.665600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áó'ôð¬‚(:å ¢êÒ³ü‹® disk.rbd2.in‚0.015625 disk.sda1.out‚0 network.sent‚500931* disk./.usagePercent‚38.763892812062124 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 28.37109375 network.recv‚179030 disk.rbd1.out‚4423.125 disk.rbd0.in‚0" disk.dm-0.out‚2182.18408203125 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚2182.18408203125 disk.sda.out‚ 28.37109375 disk.sda2.in‚0 disk.dm-0.in‚ 1226.8125 disk.nvme0c0n1.in‚ 1226.8125* proc.memory.percent‚16.852593421936035 cpu‚25.430889754528952 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.out‚ 1.9736328125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚2182.18408203125$ disk./.usageGB‚577.6100158691406 disk.sda3.out‚ 28.37109375 disk.nvme0n1.in‚ 1226.8125# memory_percent‚81.2838077545166( proc.memory.availableMB‚ 3066.4609375# proc.memory.rssMB‚ 2761.128906259) ’­‚(:‡ ž¢êÒçƘ¬ gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k–Å\û®‚(:𠞢êÒº¶½® network.recv‚182690 disk.sda1.in‚0) proc.memory.availableMB‚ 3066.47265625* proc.memory.percent‚16.852593421936035* disk./.usagePercent‚38.781396559193205 disk.rbd0.out‚0" disk.dm-0.out‚2389.28076171875 disk.dm-1.out‚ 28.640625 disk.dm-0.in‚ 1228.80078125 disk.sda3.in‚0 disk.sda3.out‚ 28.640625$ memory_percent‚81.28373622894287 disk.sda2.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1228.80078125# proc.memory.rssMB‚ 2761.12890625 cpu‚25.429590798208178 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4460.828125 network.sent‚506969$ disk./.usageGB‚577.8708343505859 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚2389.28076171875 disk.sda.in‚0 disk.sda.out‚ 28.640625 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 1.9736328125! disk.nvme0n1.in‚ 1228.80078125% disk.nvme0n1.out‚2389.28076171875,â84”¯‚(:‰ ­¢êÒÞ÷ý« gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ï°‚(:ä ­¢êÒð¿­ cpu‚25.428483884271717 disk.dm-0.in‚ 1228.8125 disk.drbd2165.out‚0 disk.sda3.out‚ 29.2109375% disk.nvme0n1.out‚3803.13330078125 disk.sda.out‚ 29.2109375$ memory_percent‚81.28042221069336* proc.memory.percent‚16.852593421936035 disk.rbd2.out‚ 1.9736328125 network.recv‚186638 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625' disk.nvme0c0n1.out‚3803.13330078125) disk./.usagePercent‚38.76780870330586$ disk./.usageGB‚577.6683654785156 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1228.8125 disk.nvme0c0n1.in‚ 1228.8125" disk.dm-0.out‚3801.25830078125# proc.memory.rssMB‚ 2761.12890625 disk.sda.in‚0 network.sent‚512389 disk.sda2.in‚0 disk.dm-1.out‚ 29.2109375' proc.memory.availableMB‚ 3067.015625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 4643.203125 disk.sda1.out‚0 disk.sda3.in‚0›«>@0±+ _wandbruntime‚565  _runtime‚565¥ÐRƲÀ ' lr-AdamW/pg1‚0.00023912294241204476 ' lr-AdamW/pg2‚0.00023912294241204476 trainer/global_step‚349 % _timestamp‚1.7843203119058294e+09 _runtime‚ 565.280459805 _step‚24µjÈÊ³Ä trainer/global_step‚349 % _timestamp‚1.7843203119058294e+09 _runtime‚ 565.280459805 _step‚24 ' lr-AdamW/pg1‚0.00023912294241204476 ' lr-AdamW/pg2‚0.00023912294241204476+‹ÒW0´+ _wandbruntime‚565  _runtime‚565‚§¢‡Â'µ¼O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 324851993088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97387624652800 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2876465 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚969855 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27312255184896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97345136193536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2986457 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97387650826240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27311887093760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚349 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97712502819328 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 323506783232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 324299171210 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2986457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2876471 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27727438298112 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97711131436032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2976835 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97711131436032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 322954572416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 324851993088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2876471 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97387624652800 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 323506783232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1864262 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 415551204352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚894407 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27727839008256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚25 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97345110765568 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97712502819328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚894403 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843203119071462e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2976835 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚969614 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 565.281931413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 415583823360Gq­k¯¶ÀO VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚894407 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27727839008256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚25 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97345110765568 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97712502819328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚894403 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843203119071462e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2976835 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚969614 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 565.281931413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 415583823360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 324851993088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97387624652800 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2876465 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚969855 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27312255184896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97345136193536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2986457 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97387650826240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27311887093760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚349 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97712502819328 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 323506783232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 324299171210 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2986457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2876471 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27727438298112 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97711131436032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2976835 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚9Qõ‚ü7711131436032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 322954572416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 324851993088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2876471 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97387624652800 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 323506783232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1864262 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 415551204352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820—AÖ0·+ _wandbruntime‚566  _runtime‚566S³zˆÌ&¸ÆM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 325222637568 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97990059298816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97665653653504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97622992543744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1869344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27389743689728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚26 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97991430689280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2884677 _runtime‚ 566.675777725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 324431818752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97991430689280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 416746173440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97946870582406 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2985313 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚972633 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27806522475008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97665627480064 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27806121772032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚972401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32611840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 324431818752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97948240609280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 325777035776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2884677 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2994949 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2884671 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 416778785280 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1869580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2884671 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 325777035776 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97665627480064 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27806121772032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚972401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32611840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 324431818752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97948240609280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 325777035776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2884677 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2994949 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2884671 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 416778785280 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1869580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2884671 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 325777035776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97622992543744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1869344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27389743689728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚26 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97991430689280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2884677 _runtime‚ 566.675777725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 324431818752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97991430689280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 416746173440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97946870582406 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2985313 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚972633 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27806522475008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 FÂpŸ½‚(:” ¼¢êҕ̈¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð o8ᾂ(:Ö ¼¢êÒ×Ö˜¯ disk.sda2.in‚0 disk.sda1.out‚0 cpu‚25.4267745172109 proc.cpu.threads‚17* disk./.usagePercent‚38.766565530719596 disk.nvme0c0n1.in‚ 1374.4375 disk.nvme0n1.in‚ 1374.4375 disk.rbd0.in‚0$ memory_percent‚81.28209114074707! proc.memory.rssMB‚ 2761.140625 network.recv‚190208 disk.rbd1.in‚0 disk.rbd2.in‚0.015625 disk.dm-1.in‚0* proc.memory.percent‚16.852664947509766 disk.dm-1.out‚29.53125 disk.sda3.in‚0 disk.sda3.out‚29.53125 disk.sda1.in‚0# disk.nvme0n1.out‚4718.572265625 disk.sda.out‚29.53125 disk.dm-0.in‚ 1374.4375 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚4718.572265625( proc.memory.availableMB‚ 3066.7421875 disk.sda2.out‚0 disk.drbd2165.out‚0$ disk./.usageGB‚577.6498413085938 disk.rbd1.out‚ 4645.453125 disk.rbd0.out‚0 disk.rbd2.out‚ 1.9736328125 disk.dm-0.out‚4718.572265625 network.sent‚517785 disk.sda.in‚0ñeZ•¿‚(:„ Ë¢êÒÄ÷ʬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚330.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L&:²ûÀ‚(:ð Ë¢êÒƒøô® disk.rbd2.in‚0.015625! disk.nvme0n1.in‚ 1465.05859375 disk.sda.out‚ 29.92578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0n1.out‚5001.8330078125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚81.28182888031006 cpu‚25.425729312647892 network.recv‚195940 disk.dm-0.in‚ 1465.05859375! disk.dm-0.out‚5001.8330078125 disk.sda3.out‚ 29.92578125& disk.nvme0c0n1.out‚5001.8330078125* proc.memory.percent‚16.852664947509766 network.sent‚565599* disk./.usagePercent‚38.758619312864305# disk./.usageGB‚577.531436920166 disk.rbd2.out‚ 1.9736328125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 3066.78515625 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 1465.05859375 disk.rbd1.in‚0! proc.memory.rssMB‚ 2761.140625 disk.rbd0.out‚0 disk.rbd1.out‚ 4852.578125 disk.rbd0.in‚0 disk.dm-1.out‚ 29.92578125 proc.cpu.threads‚17Žºr”Á‚(:‰ Ú¢êÒ‡ƒ™¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;žT÷‚(:ì Ú¢êÒÄûÀ® network.recv‚199901 disk.rbd0.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 1465.05859375& disk.nvme0c0n1.out‚5052.7724609375! proc.memory.rssMB‚ 2761.140625 network.sent‚571665 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 30.3203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 3066.7890625* proc.memory.percent‚16.852664947509766 disk.sda.in‚0 disk.rbd2.in‚0.015625! disk.dm-0.out‚5052.7724609375 disk.dm-1.out‚ 30.3203125 cpu‚25.424345893575186* disk./.usagePercent‚38.758601648344445$ disk./.usageGB‚577.5311737060547 disk.rbd1.out‚ 5023.46875 disk.rbd2.out‚ 1.9736328125$ memory_percent‚81.28180503845215 proc.cpu.threads‚17 disk.sda.out‚ 30.3203125 disk.rbd1.in‚0 disk.dm-0.in‚ 1465.05859375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1465.05859375 disk.sda1.out‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚5052.7724609375hÜKžÃ‚(:“ é¢êÒˆÍæ« gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚323.576' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.71520000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ÓÉòÄ‚(:ç é¢êҧᑮ network.sent‚576765 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd2.in‚0.015625 network.recv‚204225 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2761.140625! disk.dm-0.out‚5067.4931640625 disk.sda1.out‚0 disk.sda3.out‚ 30.6328125$ disk.nvme0n1.out‚5067.4931640625 cpu‚25.423414966590556$ disk./.usageGB‚577.5315399169922 disk.dm-0.in‚ 1466.015625 disk.sda1.in‚0& disk.nvme0c0n1.out‚5067.4931640625 disk.nvme0n1.in‚ 1466.015625! disk.nvme0c0n1.in‚ 1466.015625 disk.rbd1.out‚ 5140.796875 disk.rbd0.out‚0$ memory_percent‚81.28182888031006 proc.cpu.threads‚17 disk.sda3.in‚0) proc.memory.availableMB‚ 3066.78515625* proc.memory.percent‚16.852664947509766 disk.sda.out‚ 30.6328125 disk.rbd1.in‚0 disk.rbd2.out‚ 2.2001953125) disk./.usagePercent‚38.75862622506773 disk.drbd2165.in‚0 disk.dm-1.out‚ 30.6328125 disk.sda.in‚0rØ.ÓŸÅ‚(:” ø¢êÒ˜ò¬ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.165' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.632999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓAðí¶Æ‚(:× ø¢êÒÿу° disk.dm-0.in‚ 1466.84765625! disk.dm-0.out‚5079.4912109375 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚5079.4912109375! proc.memory.rssMB‚ 2761.140625 network.recv‚207511 disk.sda1.out‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1466.84765625$ memory_percent‚81.28182888031006 disk.sda2.in‚0) proc.memory.availableMB‚ 3066.78515625) disk./.usagePercent‚38.77764784086932 disk.rbd0.in‚0 disk.rbd2.out‚ 2.7265625 disk.sda1.in‚0 disk.dm-1.out‚31.5 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 5143.171875* proc.memory.percent‚16.852664947509766 cpu‚25.42196265218977 proc.cpu.threads‚17 network.sent‚582366 disk.sda3.out‚31.5 disk.sda.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1466.84765625$ disk.nvme0n1.out‚5079.4912109375 disk.rbd1.in‚0$ disk./.usageGB‚577.8149757385254 dišLÕ,sk.rbd2.in‚0.015625 disk.sda.out‚31.5r"þŸÇ‚(:” ‡£êÒüü…¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÁýáëÈ‚(:à ‡£êÒêÑ® network.recv‚211018 disk.rbd1.out‚ 5345.203125 disk.drbd2165.in‚0* proc.memory.percent‚16.852664947509766 disk.sda2.out‚0 network.sent‚586892 disk.rbd2.in‚0.015625 disk.sda3.in‚0& disk.nvme0c0n1.out‚5369.3017578125 disk.sda.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚5369.3017578125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚31.90625 disk.sda3.out‚31.90625 disk.sda.out‚31.90625$ disk./.usageGB‚577.7486953735352 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1538.2890625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1538.2890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1538.2890625$ memory_percent‚81.28392696380615) proc.memory.availableMB‚ 3066.44140625$ disk.nvme0n1.out‚5369.3017578125 cpu‚25.420843598450272( disk./.usagePercent‚38.7731997099619 disk.rbd2.out‚ 2.7265625! proc.memory.rssMB‚ 2761.140625î‹LÛ0É+ _wandbruntime‚645  _runtime‚645\ðÆÊÀ ' lr-AdamW/pg1‚0.00027338124361720877 ' lr-AdamW/pg2‚0.00027338124361720877 trainer/global_step‚399 % _timestamp‚1.7843203917906692e+09 _runtime‚ 645.165421057 _step‚28r¦pºÊËÄ trainer/global_step‚399 % _timestamp‚1.7843203917906692e+09 _runtime‚ 645.165421057 _step‚28 ' lr-AdamW/pg1‚0.00027338124361720877 ' lr-AdamW/pg2‚0.00027338124361720877^« Ì0Ì+ _wandbruntime‚645  _runtime‚645Ëk'Ñ'ÍËO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369191685524 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111658960113152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 369822713856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3402587 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1109334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111239199674368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31662262267392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3286771 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1021403 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3412209 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31186680426496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111608391359892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111609761386654 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111657588729856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3412209 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111287792189440 _runtime‚ 645.166676677 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 369822713856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175yìLÙÕ'ÎÏO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111657588729856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3412209 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111287792189440 _runtime‚ 645.166676677 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 369822713856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369191685524 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111658960113152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 369822713856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3402587 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1109334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111239199674368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31662262267392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3286771 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1021403 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3412209 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31186680426496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111608391359892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111609761386654 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2135812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111888566631188 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843203931873071e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3420701 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3294977 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31740941687296 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 476772755968 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 370115151770 _runtime‚ 646.562140643 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111937887983104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3420701 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1023943 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31263800840192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111565795016704 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3411065 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚399 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111517106880512 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111517081452544 _step‚30 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3294971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111565795016704 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 370747749376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 476738046976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111565768843264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111517106880512 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111517081452544 _step‚30 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3294971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111565795016704 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 370747749376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 476738046976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111565768843264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2135812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111888566631188 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843203931873071e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3420701 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3294977 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31740941687296 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 476772755968 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 370115151770 _runtime‚ 646.562140643 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111937887983104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3420701 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1023943 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31263800840192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992í¿—†0Ò+ _wandbruntime‚646  _runtime‚646/IHTÓú 2loss/SRqqt2/assignment_loss‚1.7927765989619675 .loss/FBt1/detection_loss‚5.639738402767283 % _timestamp‚1.7843203932701344e+09 _runtime‚ 646.644644503 /loss/FRt1/assignment_loss‚4.275795670411697 /loss/FRt2/assignment_loss‚4.577939120323761 .loss/FBt2/detection_loss‚5.048850131400935 &loss/total_loss‚29.213628990638284 trainer/global_step‚399 2loss/SRqqt1/assignment_loss‚1.5164407279513115 1loss/FBt1/assignment_loss‚0.13617036808750163 0loss/FBt2/assignment_loss‚0.3678825058897388 /loss/FRt2/detection_loss‚0.4362176734025327 epoch‚0 _step‚31 /loss/FRt1/detection_loss‚1.4474924193018575 1loss/SRqqt1/detection_loss‚0.8528695434021715 1loss/SRqqt2/detection_loss‚3.1214558287375245˜ "+Ôþ 2loss/SRqqt2/assignment_loss‚1.7927765989619675 .loss/FBt1/detection_loss‚5.639738402767283 % _timestamp‚1.7843203932701344e+09 _runtime‚ 646.644644503 /loss/FRt1/assignment_loss‚4.275795670411697 /loss/FRt2/assignment_loss‚4.577939120323761 .loss/FBt2/detection_loss‚5.048850131400935 &loss/total_loss‚29.213628990638284 trainer/global_step‚399 2loss/SRqqt1/assignment_loss‚1.5164407279513115 1loss/FBt1/assignment_loss‚0.13617036808750163 0loss/FBt2/assignment_loss‚0.3678825058897388 /loss/FRt2/detection_loss‚0.4362176734025327 epoch‚0 _step‚31 /loss/FRt1/detection_loss‚1.4474924193018575 1loss/SRqqt1/detection_loss‚0.8528695434021715 1loss/SRqqt2/detection_loss‚3.1214558287375245n|¬C”Õ‚(:‰ –£êÒÍ玬 gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Š0ÿÖ‚(:ô –£êÒ˜ñͯ disk.rbd2.in‚0.015625 disk.sda1.out‚0) proc.memory.availableMB‚ 2990.58203125" proc.memory.rssMB‚ 2836.7109375' disk.nvme0c0n1.out‚5377.93115234375 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 5534.828125 disk.rbd0.in‚0 disk.rbd2.out‚ 2.78564453125) proc.memory.percent‚17.31390953063965! disk.nvme0n1.in‚ 1539.24609375 disk.sda2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚214017$ memory_percent‚81.74693584442139 cpu‚25.469220861617877 disk.drbd2165.in‚0 network.sent‚592527" disk.dm-0.out‚5377.93115234375$ disk./.usageGB‚577.7472076416016% disk.nvme0n1.out‚5377.93115234375) disk./.usagePercent‚38.77309986702354 disk.sda3.out‚ 32.36328125# disk.nvme0c0n1.in‚ 1539.24609375 disk.sda.out‚ 32.36328125 disk.rbd1.in‚0 disk.dm-0.in‚ 1539.24609375 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 32.36328125 disk.sda3.in‚0 disk.rbd0.out‚0µ ö·@×b; particle/accuracy_5_of_5"trainer/global_step2:JP"ƒÿƒJØbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÍ… µ;Ùb6 jet/accuracy_5_of_5"trainer/global_step2:JP©ÈJÚbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPRÆ°)NÛbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP5_£JÜbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPEÌ.NÝbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP!3JÞbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP5Û9;ßb6 jet/accuracy_1_of_4"trainer/global_step2:JP”yHàbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP¢JAJábE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPŸt:HâbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPبJãbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPùÐHäbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP®åñâ@åb; particle/accuracy_2_of_3"trainer/global_step2:JPHí«ì@æb; particle/accuracy_2_of_4"trainer/global_step2:JP(¦¸JçbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP»}HèbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPò˜£Í;éb6 jet/accuracy_1_of_5"trainer/global_step2:JP1%%†NêbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP$"JëbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP2/ØñHìbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÐüô(JíbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP™&g‹NîbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPviHïbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP h)xGðbB validation_average_jet_accuracy"trainer/global_step2:JP¤ë©THñbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPQÅ7gHòbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP$µóJóbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPò´“ôJôbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPúKÉ4NõbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŸàgVHöbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP­‰ãH÷bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÎ9F´HøbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPï4ÄNùbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPkóü²@úb; particle/accuracy_2_of_5"trainer/global_step2:JP@ÙmJûbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP0™¶zHübC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP6ɧINýbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP±ì³ÓNþbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP¥la:@ÿb; particle/accuracy_1_of_1"trainer/global_step2:JPî™Z±J€bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP˜y/,JbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÁAwH‚bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÌÉä@ƒb; particle/accuracy_3_of_4"trainer/global_step2:JP…H>:<„b7 particle/specificity"trainer/global_step2:JPs!‘ø@…b; particle/accuracy_3_of_5"trainer/global_step2:JPïå“åN†bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPŸOON‡bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPSQ3HˆbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP`ÓjeH‰bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPtö]HŠbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP8›è%J‹bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP±œ„BJŒbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP©` ÑNbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP6~ªHŽbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP’Zô;b6 jet/accuracy_1_of_2"trainer/global_step2:JPñ¹G³;b6 jet/accuracy_1_of_3"trainer/global_step2:JPC¨Œ H‘bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÜ[°H’bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPñ)n‰J“bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPë74PN”bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÆÂH•bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÓ?Á¾H–bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPAQENJ—bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP^nH˜bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPYtèçH™bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPW±®NšbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÙ›N›bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP£Ì]@œb; particle/accuracy_4_of_4"trainer/global_step2:JPJì0NbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÜ‹^ŒHžbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPáçHŸbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¢~[ÊJ bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPð‡›N¡bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPGÃKæH¢bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPlGN£bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÎ,‡ÜJ¤bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPG©­H¥bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPz®|ËJ¦bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP!}ÏJ§bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP Q=;¨b6 jet/accuracy_2_of_4"trainer/global_step2:JP"ûb¦J©bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPó\bHªbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP ^]ÜJ«bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPósDKN¬bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¼ÕK H­bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPþ¡œœH®bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPñ'kJ¯bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŽµÐüJ°bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPRBlæH±bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP•›ÅuN²bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPˤ³áH³bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPIPlN´bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP­*4‹HµbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP£9H¶bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¶6%žJ·bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPüŽF5J¸bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPhy¦J¹bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÏjHºbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPN'ÿ‰N»bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP²vËN¼bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÔÕ 4J½bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP2vH¾bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPƒo±®J¿bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP—ú‡ÊJÀbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP™üwJÁbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÌÿdJÂbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPðhJÃbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP jÿNÄbI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP8õÜcNÅbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPªÜ“JÆbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPõ4HÇbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP]»¬HNÈbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP'u×äHÉbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPzîëKJÊbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPa0¡ÝJËbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPevŒNÌbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP?rNÍbI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP÷Å)NÎbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP´øcNÏbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP·BXfJÐbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¾O/NÑbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP(¡áJÒbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP2Ò…ÊJÓbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP7"Ø;Ôb6 jet/accuracy_3_of_3"trainer/global_step2:JPDµÙ/JÕbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÒ$#HNÖbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP‚˜XbH×bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP‰jº4HØbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÙãNÙbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP2²ÚNÚbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÚ)àgJÛbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPœ­¦”JÜbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPáñ4fHÝbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP‹ó%nJÞbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPEDZñ;ßb6 jet/accuracy_4_of_4"trainer/global_step2:JP¼Î**JàbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPþɆèJábE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPªˆþCJâbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŒª³öNãbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPToèÏHäbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPEñõJåbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPºónHæbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP ˆÀ³HçbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPQ”-HèbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPu ¾>HébC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP0|PNêbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPy@ˆ JëbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPRšÌJìbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPlkWéHíbC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP‚ÐrJîbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPJ™ÓJïbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!iÆJðbE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP•¬©èJñbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPX(hHòbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÞÂ$wNóbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÇ~!HôbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÔHÍJõbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPž7/­JöbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPs*36J÷bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPú— t@øb; particle/accuracy_1_of_5"trainer/global_step2:JP³†ÍµHùbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPd™%‡JúbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPûò9ENûbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPˆ×á8JübE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÿèƒ8HýbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¡ƒ( JþbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP°,ãNÿbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPI„÷J€bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPYžNJbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPµÚz\N‚bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPK¾kHƒbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP@÷ ÜH„bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP‚ÈUJ…bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP9t&J†bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÔU û;‡b6 jet/accuracy_2_of_5"trainer/global_step2:JP‰Ke÷HˆbC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPûròJ‰bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPs7ÿ§NŠbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPZà%N‹bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPc%‹YJŒbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ª!×JbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP·¢‹ØNŽbI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP;ů;b6 jet/accuracy_2_of_3"trainer/global_step2:JP.¸NbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÑs~N‘bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPwš"5H’bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÆŠ»2;“b6 jet/accuracy_3_of_5"trainer/global_step2:JPüŽXH”bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÏ)H•bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPDJã<–b7 particle/sensitivity"trainer/global_step2:JPJU;—b6 loss/val_total_loss"trainer/global_step2:JP²¦ç%H˜bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPý®æ%H™bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÃõøÒJšbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPb&T°N›bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPE“ JœbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPivËëNbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP 6ÁHžbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPX[ÇNŸbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPV†^ÂN bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÄY2N¡bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPúp<úH¢bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP‹Ž™;£b6 jet/accuracy_4_of_5"trainer/global_step2:JPÏã4:@¤b; particle/accuracy_4_of_5"trainer/global_step2:JPËl/ÀH¥bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP üÝdN¦bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPïŒ]H§bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPMHÎJ¨bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPï½)“J©bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP&ÛEAHªbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPφíUH«bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP–ÍbY@¬b; particle/accuracy_1_of_4"trainer/global_step2:JPS¡·H­bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÒã”J®bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP B‰¬J¯bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP 6<öJ°bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÖ+ÙJ±bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÆäVN²bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP ½v{N³bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÉa@´b; particle/accuracy_1_of_2"trainer/global_step2:JPìò7@µb; particle/accuracy_1_of_3"trainer/global_step2:JPã¡‚›H¶bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP}~ø~J·bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPàÞŽÿH¸bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPýÉtDJ¹bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÅ>5¨JºbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPBÂʆH»bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÊÑýN¼bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP•ÐÑJ½bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPVöÂH¾bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP0N¿bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPó­£ÆHÀbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÀz˜HÁbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPþÍhÐ@Âb; particle/accuracy_3_of_3"trainer/global_step2:JPgü@NNÃbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÒÀ;"JÄbE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰ÊM'NÅbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP2<ƒ§JÆbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÌ=ÞWJÇbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP›•lHÈbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP±,‘¼JÉbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP·B¨HÕbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP6oÄNÖbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP´a}ÍJ×bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP ³'"NØbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP!Ä&IJÙbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP9˜JÚbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP/ô<¾;Ûb6 jet/accuracy_2_of_2"trainer/global_step2:JPdýSJÜbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP]˵JÝbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP±ãÄf9Þb4 particle/accuracy"trainer/global_step2:JP-‘ÒÐHßbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP[#ÒJàbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPdA)JábE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPrL3JâbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPXq¢;ãb6 jet/accuracy_1_of_1"trainer/global_step2:JP‡ÅÔ€HäbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP4ÈuäJåbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPSNæbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP’ÄRJçbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPš¸ª¾HèbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÎ3Í';éb6 jet/accuracy_3_of_4"trainer/global_step2:JPer`JêbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPnîôJëbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPïýu*HìbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP2RêJíbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP‰?ì(8îb3 particle/f_score"trainer/global_step2:JPé– HïbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP–ró¤HðbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPôºKqNñbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP:…{HòbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP†Š» NóbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP=>•µJôbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP¶ÔüHõbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÎ2VfHöbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP`õÂ0÷+ _wandbruntime‚673  _runtime‚673ÑŸÔ×)=ø£z 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1864042431116104 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 trainer/global_step‚399 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008118863217532635 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919494390487671 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.528323233127594 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8836002349853516 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9620487689971924 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9852418303489685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0028453022241592407 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9657124876976013 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9969338178634644 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.009456655010581017 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9833347797393799 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 _runtime‚ 673.232440581 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.11150838434696198 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.008220599964261055 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_3‚0.08788588643074036 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 jet/accuracy_3_of_5‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1300269365310669 +particle/sensitivity‚0.5176053047180176 *loss/val_total_loss‚27.762895323385205 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972979426383972 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt*SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.09127786010503769 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.01587301678955555 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0572916679084301 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.011407749727368355 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19412371516227722 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.027349283918738365 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.011336639523506165 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.06699194759130478 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.05195482447743416 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1450149267911911 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3965378701686859 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.011118698865175247 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08214765787124634 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18778134882450104 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.43518519401550293 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt1SRqqt2FBt/FRt_purity‚0 9 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.054999999701976776 particle/accuracy_3_of_3‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.09271959960460663 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.11319229751825333 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0013076341710984707 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 particle/accuracy_2_of_2‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9901124835014343 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13727228343486786 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.010847823694348335 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.402442067861557 +jet/accuracy_2_of_2‚0.03937923535704613 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.002899913350120187 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.09735206514596939 $particle/accuracy‚0.762451171875 9 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.024790698662400246 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9962860941886902 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.15144868195056915 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.07847445458173752 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06710152328014374 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.09951513260602951 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12064038962125778 -jet/accuracy_3_of_4‚0.0031746034510433674 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.10431086272001266 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 (particle/f_score‚0.48342958092689514 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.010186972096562386 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11714302003383636 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06620867550373077 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0416666679084301 particle/accuracy_5_of_5‚0 ="Purity/1FRt1SRqqt*FBt/event_purity‚0.00015069318760652095 jet/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.05261782184243202 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10973791778087616 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.11030246317386627 *jet/accuracy_1_of_4‚0.4970189929008484 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08474428206682205 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9861111044883728 ="Purity/1FRt1SRqqt0FBt/event_purity‚0.00015511090168729424 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 _step‚32 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.43518519401550293 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9861111044883728 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.10121913254261017 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.1089121624827385 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3956981301307678 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.00881417840719223 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11612961441278458 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06486475467681885 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9751683473587036 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.39438170194625854 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.14223641157150269 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12225925177335739 /particle/accuracy_3_of_4‚0.9859755635261536 +particle/specificity‚0.8294171690940857 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.05257236212491989 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 9 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.021068064495921135 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.10064378380775452 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9389428496360779 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05942189693450928 *jet/accuracy_1_of_2‚0.3283150792121887 *jet/accuracy_1_of_3‚0.5009554624557495 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944952130317688 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9877106547355652 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.09127786010503769 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957554936408997 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6494237780570984 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.04684944823384285 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10620751976966858 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.9428380131721497 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.011904762126505375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9929211139678955 ("Purity/1FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.059157151728868484 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6948586702346802 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06013447791337967 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.14621445536613464 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 +jet/accuracy_2_of_4‚0.07184652239084244 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7447980046272278 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859740138053894 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9962860941886902 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3125 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10620751976966858 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9880766272544861 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.011904762126505375 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.07362629473209381 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703880310059 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.08860084414482117 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05850100517272949 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7495865821838379 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.10121698677539825 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 $ _timestamp‚1.784320419856861e+09 9 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.033046871423721313 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9938241839408875 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.11562246084213257 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.011231840588152409 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.09211168438196182 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08611011505126953 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_3‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9694714546203613 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9824402332305908 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 epoch‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.40649154782295227 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.20690162479877472 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4112405776977539 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.14956343173980713 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.179733544588089 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.1082811951637268 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9702380895614624 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.01587301678955555 & Purity/*FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01153632253408432 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.10330402106046677 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0013379318406805396 9 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.032317645847797394 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.11030246317386627 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.15038935840129852 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.005366969853639603 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.011336639523506165 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969ŸÂëb7ù§z :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06710152328014374 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.09951513260602951 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12064038962125778 -jet/accuracy_3_of_4‚0.0031746034510433674 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.10431086272001266 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 (particle/f_score‚0.48342958092689514 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.010186972096562386 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11714302003383636 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06620867550373077 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0416666679084301 particle/accuracy_5_of_5‚0 ="Purity/1FRt1SRqqt*FBt/event_purity‚0.00015069318760652095 jet/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.05261782184243202 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10973791778087616 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.11030246317386627 *jet/accuracy_1_of_4‚0.4970189929008484 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08474428206682205 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9861111044883728 ="Purity/1FRt1SRqqt0FBt/event_purity‚0.00015511090168729424 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 _step‚32 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.43518519401550293 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9861111044883728 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.10121913254261017 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.1089121624827385 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3956981301307678 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.00881417840719223 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11612961441278458 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06486475467681885 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9751683473587036 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.39438170194625854 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.14223641157150269 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12225925177335739 /particle/accuracy_3_of_4‚0.9859755635261536 +particle/specificity‚0.8294171690940857 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.05257236212491989 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 9 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.021068064495921135 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.10064378380775452 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9389428496360779 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05942189693450928 *jet/accuracy_1_of_2‚0.3283150792121887 *jet/accuracy_1_of_3‚0.5009554624557495 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944952130317688 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9877106547355652 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.09127786010503769 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957554936408997 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6494237780570984 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.04684944823384285 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10620751976966858 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.9428380131721497 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.011904762126505375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9929211139678955 ("Purity/1FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.059157151728868484 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6948586702346802 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06013447791337967 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.14621445536613464 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 +jet/accuracy_2_of_4‚0.07184652239084244 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7447980046272278 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859740138053894 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9962860941886902 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3125 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10620751976966858 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9880766272544861 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.011904762126505375 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.07362629473209381 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703880310059 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.08860084414482117 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05850100517272949 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7495865821838379 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.10121698677539825 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 $ _timestamp‚1.784320419856861e+09 9 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.033046871423721313 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9938241839408875 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.11562246084213257 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.011231840588152409 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.09211168438196182 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08611011505126953 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_3‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9694714546203613 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9824402332305908 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 epoch‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.40649154782295227 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.20690162479877472 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4112405776977539 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.14956343173980713 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.179733544588089 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.1082811951637268 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9702380895614624 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.01587301678955555 & Purity/*FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01153632253408432 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.10330402106046677 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0013379318406805396 9 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.032317645847797394 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.11030246317386627 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.15038935840129852 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.005366969853639603 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.011336639523506165 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1864042431116104 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 trainer/global_step‚399 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008118863217532635 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919494390487671 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.528323233127594 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8836002349853516 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9620487689971924 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9852418303489685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0028453022241592407 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9657124876976013 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9969338178634644 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.009456655010581017 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9833347797393799 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 _runtime‚ 673.232440581 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.11150838434696198 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.008220599964261055 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_3‚0.08788588643074036 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 jet/accuracy_3_of_5‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1300269365310669 +particle/sensitivity‚0.5176053047180176 *loss/val_total_loss‚27.762895323385205 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972979426383972 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt*SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.09127786010503769 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.01587301678955555 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0572916679084301 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.011407749727368355 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19412371516227722 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.027349283918738365 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.011336639523506165 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.06699194759130478 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.05195482447743416 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1450149267911911 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3965378701686859 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.011118698865175247 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08214765787124634 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18778134882450104 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.43518519401550293 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt1SRqqt2FBt/FRt_purity‚0 9 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.054999999701976776 particle/accuracy_3_of_3‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.09271959960460663 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("PurityŠä’Ë/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.11319229751825333 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0013076341710984707 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 particle/accuracy_2_of_2‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9901124835014343 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13727228343486786 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.010847823694348335 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.402442067861557 +jet/accuracy_2_of_2‚0.03937923535704613 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.002899913350120187 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.09735206514596939 $particle/accuracy‚0.762451171875 9 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.024790698662400246 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9962860941886902 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.15144868195056915 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.07847445458173752 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 ]o>e“ú‚(:ˆ ¥£êÒ—팭 gpu.0.gpu‚22.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÝ0ðû‚(:å ¥£êÒÈÏâ® disk.sda1.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1539.24609375 network.sent‚639316 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 5392.66796875! disk.nvme0n1.in‚ 1539.24609375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.sda3.out‚ 32.953125" disk.nvme0n1.out‚ 5392.66796875 disk.sda2.in‚0 disk.rbd1.out‚ 5694.453125) proc.memory.percent‚22.96454906463623) proc.memory.availableMB‚ 2063.99609375 cpu‚25.518526620247375 proc.cpu.threads‚17) disk./.usagePercent‚38.77311957960367 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 network.recv‚218453 disk.dm-0.in‚ 1539.24609375 disk.dm-0.out‚ 5392.66796875 disk.dm-1.out‚ 32.953125 disk.sda.out‚ 32.953125$ memory_percent‚87.40236759185791# proc.memory.rssMB‚ 3762.51171875# disk./.usageGB‚577.747501373291 disk.rbd0.in‚0 disk.rbd2.out‚ 2.78564453125 disk.sda1.in‚03äV8žü‚(:“ ´£êÒ¸ïç« gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îàbëôý‚(:é ´£êÒ Ô® disk.rbd2.out‚ 2.8525390625$ disk./.usageGB‚579.6409530639648% disk.nvme0n1.out‚7339.91552734375( proc.memory.availableMB‚ 1948.2109375 disk.dm-0.in‚ 1540.078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 1540.078125 disk.rbd1.in‚0$ memory_percent‚88.10906410217285 proc.cpu.threads‚17 network.sent‚696332* disk./.usagePercent‚38.900190711276586 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚7339.91552734375 disk.sda2.out‚0 network.recv‚222674 proc.memory.rssMB‚ 2841.9375 disk.rbd2.in‚0.015625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 33.18359375! disk.nvme0c0n1.in‚ 1540.078125 disk.sda2.in‚0 disk.rbd1.out‚ 5784.6875 disk.rbd0.out‚0 cpu‚25.514999303314628 disk.dm-1.out‚ 33.18359375 disk.sda.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.345809936523438" disk.dm-0.out‚7339.91552734375 disk.sda.out‚ 33.18359375Ø-ó”þ‚(:‰ ãêÒÏÉê« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö4–íêÿ‚(:ß ãêÒÍÞº® disk.sda3.in‚0" disk.nvme0c0n1.in‚ 1789.4921875$ memory_percent‚88.11092376708984 cpu‚25.5116219285664# disk.nvme0n1.out‚7352.966796875 disk.rbd1.in‚0 disk.dm-1.out‚ 34.8828125 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 5936.03125) disk./.usagePercent‚38.90019224732179$ disk./.usageGB‚579.6409759521484 disk.nvme0n1.in‚ 1789.4921875 disk.sda2.out‚0 disk.sda3.out‚ 34.8828125& proc.memory.availableMB‚ 1947.90625 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 2.8525390625 proc.cpu.threads‚17 disk.sda.out‚ 34.8828125% disk.nvme0c0n1.out‚7352.966796875 proc.memory.rssMB‚ 2841.9375 network.sent‚701580 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚7352.966796875 disk.drbd2165.out‚0* proc.memory.percent‚17.345809936523438 disk.rbd0.in‚0 network.recv‚226410 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1789.4921875±’ˆœ€‚(:‘ Ò£êÒüœÇ¬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚326.236' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.24719999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gëà¿Õ‚(:Ê Ò£êÒþÐè®$ memory_percent‚88.10949325561523 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 36.2109375 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚6007.5' proc.memory.availableMB‚ 1948.140625 proc.cpu.threads‚17 disk.dm-0.out‚7360.75 disk.sda3.out‚ 36.2109375 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1789.4921875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 1789.4921875 disk.sda.out‚ 36.2109375 disk.rbd0.in‚0 cpu‚25.509348324990686 network.sent‚706725 disk.rbd2.out‚ 2.9150390625 disk.sda1.out‚0 disk.nvme0c0n1.out‚7360.75* proc.memory.percent‚17.345809936523438* disk./.usagePercent‚38.900157430297135$ disk./.usageGB‚579.6404571533203 disk.rbd2.in‚0.015625 disk.nvme0n1.in‚ 1789.4921875 proc.memory.rssMB‚ 2841.9375 network.recv‚229629 disk.dm-1.in‚0 disk.nvme0n1.out‚7360.75 disk.sda2.out‚0ì“çeŸ‚‚(:” á£êÒ†©Š¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q²1䃂(:Ù á£êÒèŽù® network.sent‚711809$ disk.nvme0n1.out‚7387.8720703125 disk.sda.out‚ 39.609375 disk.rbd0.in‚0 disk.rbd2.out‚ 2.9150390625 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚38.90077824856767 disk.sda3.out‚ 39.609375 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚88.11259269714355$ disk./.usageGB‚579.6497077941895 disk.nvme0c0n1.in‚ 1790.4375& disk.nvme0c0n1.out‚7387.8720703125 disk.rbd2.in‚0.015625 proc.memory.rssMB‚ 2841.9375 proc.cpu.threads‚17 disk.sda2.in‚0! disk.dm-0.out‚7387.8720703125 disk.nvme0n1.in‚ 1790.4375 disk.dm-0.in‚ 1790.4375 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1947.6328125 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.345809936523438 disk.sda3.in‚0 disk.dm-1.out‚ 39.609375 cpu‚25.506193543055687 network.recv‚233204 disk.sda.in‚0 disk.rbd1.out‚ 6133.3125f¸î]“„‚(:ˆ ð£êÒ¦ñѬ gpu.0.gpu‚15.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ~Ÿ(ï…‚(:ä ð£êÒÍÞæ® disk.dm-1.out‚42.5625( disk./.usagePercent‚38.6920952430769 proc.memory.rssMB‚ 2841.9375 disk.sda.in‚0 disk.drbd2165.in‚0$ memory_percent‚88.11113834381104* proc.memory.percent‚17.345809936523438$ disk./.usageGB‚576.5401802062988 disk.rbd2.out‚ 3.3935546875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 network.sent‚715903 network.recv‚235665! disk.nvme0n1.in‚ 1791.50390625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚7440.96826171875 disk.sda2.out‚0 disk.rbd1.out‚ 6376.078125 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1791.50390625 cpu‚25.503824935672363 disk.sda.out‚42.5625 disk.sda3.out‚42.5625' disk.nvme0c0n1.out‚7440.96826171875) proc.memory.availableMB‚ 1947.87109375 disk.dm-0.in‚ 1791.50390625" disk.dm-0.out‚7440.96826171875Z^Þ0†+ _wandbruntime‚755  _runtime‚755ôZNć¾ % _timestamp‚1.7843205025102737e+09 _runtime‚ 755.884923012 _step‚33 & lr-AdamW/pg1‚0.0002740621832145334 & lr-AdamW/pg2‚0.0002740621832145334 trainer/global_step‚449øCvóȈ trainer/global_step‚449 % _timestamp‚1.7843205025102737e+09 _runtime‚ 755.884923012 _step‚33 & lr-AdamW/pg1‚0.0002740621832145334 & lr-AdamW/pg2‚0.0002740621832145334!Š? æ0‰+ _wandbruntime‚755  _runtime‚755ëçI¼Ò'ŠÌO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129015786053632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚34 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129381825472967 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3897003 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1280552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128959197515776 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3829077 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 423326496768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129015812227072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 421257928909 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35652121265664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3887032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 543006111232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129380455444685 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3897003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 422602529223 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 421981108224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129015812227072 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1157472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1157467 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129015786053632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129437767161856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3887032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129439138723840 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129437767161856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 421981108224 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35651718637056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 755.889355613 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3829071 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129439138723840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3829071 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843205025114815e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 542971573760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 423326496768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3829077 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2438024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2437771 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35108747063296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175/óóÖ'‹ÐO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2438024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2437771 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35108747063296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129015786053632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚34 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129381825472967 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3897003 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1280552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128959197515776 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3829077 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 423326496768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129015812227072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 421257928909 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35652121265664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3887032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 543006111232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129380455444685 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3897003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 422602529223 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 421981108224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129015812227072 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1157472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1157467 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129015786053632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129437767161856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3887032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129439138723840 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129437767161856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 421981108224 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35651718637056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 755.889355613 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3829071 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129439138723840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3829071 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843205025114815e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 542971573760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 423326496768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3829077 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648"çxvk0Œ+ _wandbruntime‚757  _runtime‚757·ÂF¡ÕÖM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 423525995581 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 422906143744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35186601562112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 544209453568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129237104721920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 424251539456 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1160010 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129716692927488 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3837277 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35186233470976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34530304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚449 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 424251539456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35730811015680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3837283 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1283307 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1160015 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3905495 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129293786783744 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 422906143744 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7732787 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2443071 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1283061 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3895510 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129660630717501 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚35 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129293786783744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3905495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129293812957184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129293812957184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3895510 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129718064496640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129718064496640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544174923264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 757.271957176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35730408394240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129660630717501 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚35 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129293786783744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3905495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129293812957184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129293812957184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3895510 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129718064496640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129718064496640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544174923264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 757.271957176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35730408394240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 424251539456 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1160010 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129716692927488 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3837277 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35186233470976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34530304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚449 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 424251539456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35730811015680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3837283 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1283307 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1160015 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3905495 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129293786783744 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 422906143744 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7732787 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2443071 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1283061 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3895510 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603003356160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4239377 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143384119598080 YHDeviceStatsMonitor.on_train_batch_start/allocated¨ï©e _bytes.small_pool.freed‚ 468297038848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1419729 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142915822559232 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4312784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39586438894592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 467495042017 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 469642427392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4239371 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2704351 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142853311852544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 $ _timestamp‚1.784320581176662e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4322755 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143384119598080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1284622 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143384119598080 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 468297038848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1419729 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142915822559232 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4312784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39586438894592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 467495042017 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 469642427392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4239371 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2704351 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142853311852544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 $ _timestamp‚1.784320581176662e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4322755 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143384119598080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1284622 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603003356160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4239377&åÕúw0¢+ _wandbruntime‚835  _runtime‚835TÏåÛ&£ÕM F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843205825626683e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 470567470080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2709352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 470567470080 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39060921946112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 469763108689 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39060553854976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1287160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4331247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143193823289344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39665117461504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚39 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 469222074368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143193849462784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4321262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143193849462784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143193823289344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143663045363712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1422458 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604195515392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4247583 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1422192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604158887936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4247577 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 469222074368 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4247583 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39664712742912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚ 835.93764212 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4321262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143664416932864 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4247577 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143600956739409 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143131193630720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2709623 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚499®ó‚ ñ¤ÙM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143664416932864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39664712742912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚ 835.93764212 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4321262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143664416932864 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4247577 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143600956739409 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143131193630720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2709623 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚499 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843205825626683e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 470567470080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2709352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 470567470080 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39060921946112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 469763108689 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39060553854976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1287160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4331247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143193823289344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39665117461504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚39 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 469222074368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143193849462784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4321262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [GDeviceStatsMonitor.on_train_batch_end/active_bytesr;ÏLî.large_pool.allocated‚143193849462784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143193823289344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143663045363712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1422458 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604195515392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4247583 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1422192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604158887936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4247577 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 469222074368 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4247583 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 913.473379485 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157331843596288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42857755922432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156815885238272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 515076755439 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4748507 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157330472034304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157262477516783 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1411772 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 662550407168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157261107488501 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 514612969472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156815885238272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2969753 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 662513772544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 515958358016 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4738536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156747400761344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156815859064832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4649677 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 913.473379485 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157331843596288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42857755922432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156815885238272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 515076755439 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4748507 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157330472034304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157262477516783 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1411772 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 662550407168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157261107488501 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 514612969472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156815885238272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2969753 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 662513772544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 515958358016 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4738536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156747400761344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156815859064832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4649677 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157093859794944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 516883400704 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4657877 XGDevicez–ËTE$StatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 516883400704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 515538004992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4756999 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157609397799936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2974735 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43598575179776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 514655621371 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 515538004992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157025257111552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4657883 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43598979898368 % _timestamp‚1.7843206614790819e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42934874238976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157093859794944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157610769369088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157093885968384 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚549 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4657877 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157541282761317 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157539912732923 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1414315 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1560690 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚ 914.853862047 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157610769369088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚43 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2975005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157025282539520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4657883°ô3à&ºÚM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157610769369088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚43 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2975005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157025282539520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4657883 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42935242330112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157093859794944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 516883400704 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4657877 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 516883400704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 515538004992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4756999 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157609397799936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2974735 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43598575179776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 514655621371 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 515538004992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157025257111552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4657883 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43598979898368 % _timestamp‚1.7843206614790819e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42934874238976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157093859794944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157610769369088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157093885968384 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚549 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4657877 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157541282761317 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157539912732923 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1414315 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1560690 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚ 914.853862047 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216+Vm‰0»+ _wandbruntime‚914  _runtime‚914Ìë©iÿ¼ù 2loss/SRqqt2/assignment_loss‚1.6458762089885373 /loss/FRt2/detection_loss‚0.5508093694548676 0loss/SRqqt1/detection_loss‚0.660697232274027 &loss/total_loss‚21.366495797067444 _runtime‚ 914.939913854 1loss/FBt2/assignment_loss‚0.32701617565045243 1loss/SRqqt2/detection_loss‚2.4228161547434093 .loss/FBt2/detection_loss‚3.551223586367573 epoch‚1 $ _timestamp‚1.784320661565576e+09 /loss/FRt1/assignment_loss‚3.479160050811379 .loss/FBt1/detection_loss‚1.719820043997676 _step‚44 0loss/FRt2/assignment_loss‚3.9626560928349828 2loss/SRqqt1/assignment_loss‚1.5030460553049014 1loss/FBt1/assignment_loss‚0.09595830514468155 .loss/FRt1/detection_loss‚1.447416521494956 trainer/global_step‚549§»&½ý 0loss/SRqqt1/detection_loss‚0.660697232274027 &loss/total_loss‚21.366495797067444 _runtime‚ 914.939913854 1loss/FBt2/assignment_loss‚0.32701617565045243 1loss/SRqqt2/detection_loss‚2.4228161547434093 .loss/FBt2/detection_loss‚3.551223586367573 epoch‚1 $ _timestamp‚1.784320661565576e+09 /loss/FRt1/assignment_loss‚3.479160050811379 .loss/FBt1/detection_loss‚1.719820043997676 _step‚44 0loss/FRt2/assignment_loss‚3.9626560928349828 2loss/SRqqt1/assignment_loss‚1.5030460553049014 1loss/FBt1/assignment_loss‚0.09595830514468155 .loss/FRt1/detection_loss‚1.447416521494956 trainer/global_step‚549 2loss/SRqqt2/assignment_loss‚1.6458762089885373 /loss/FRt2/detection_loss‚0.5508093694548676,¡K†pŸ¾‚(:” •¥êÒþø㬠gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vlö¹¿‚(:õ •¥êÒ悯# disk.nvme0c0n1.in‚ 2061.06640625% proc.memory.availableMB‚ 10.984375* proc.memory.percent‚17.345833778381348 network.recv‚274403! disk.dm-0.out‚7628.0439453125 disk.drbd2165.in‚0) disk./.usagePercent‚38.69202228092964& disk.nvme0c0n1.out‚7628.0439453125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 2061.06640625 disk.sda3.out‚ 65.5546875 disk.sda.out‚ 65.5546875 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚99.93295669555664 network.sent‚887191$ disk.nvme0n1.out‚7628.0439453125 disk.rbd2.in‚0.015625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 7739.78515625 disk.rbd2.out‚ 4.3115234375 disk.dm-1.out‚ 65.5546875! disk.nvme0n1.in‚ 2061.06640625 disk.sda.in‚0 disk.rbd1.in‚ 0.00390625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2841.94140625 cpu‚25.48026488686359$ disk./.usageGB‚576.5390930175781ó£_ï”À‚(:‰ ¤¥êÒ»ò¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹±ò´Á‚(:ù ¤¥êÒÉ…–® network.recv‚278771 disk.rbd2.out‚ 4.3896484375" disk.dm-0.out‚7638.25634765625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚7638.25634765625! disk.nvme0n1.in‚ 2061.06640625% disk.nvme0n1.out‚7638.25634765625 disk.sda.in‚0 disk.sda.out‚ 66.3984375* proc.memory.percent‚17.345857620239258 proc.cpu.threads‚17* disk./.usagePercent‚38.692182029631006 disk.rbd0.out‚0 disk.dm-1.out‚ 66.3984375# disk.nvme0c0n1.in‚ 2061.06640625 disk.sda2.out‚0 network.sent‚949395 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.00390625 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.47878913996025 disk.dm-0.in‚ 2061.06640625 disk.sda1.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 11.1796875" proc.memory.rssMB‚ 2841.9453125 disk.drbd2165.in‚0$ disk./.usageGB‚576.5414733886719 disk.rbd2.in‚0.015625 disk.sda3.out‚ 66.3984375 disk.sda3.in‚0 disk.rbd1.out‚ 8020.86328125$ memory_percent‚99.93176460266113B'h蔂(:‰ ³¥êÒ…Õé« gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚319.073' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.8146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?õTøÂ(:í ³¥êÒÎÁœ® disk.drbd2165.out‚0$ memory_percent‚99.93045330047607 network.recv‚282328 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚ 68.890625 disk.rbd2.in‚0.015625 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 2153.65625' disk.nvme0c0n1.out‚7657.74755859375* proc.memory.percent‚17.345857620239258 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚ 2153.65625" disk.dm-0.out‚7657.74755859375% disk.nvme0n1.out‚7657.74755859375" proc.memory.rssMB‚ 2841.9453125 disk.rbd1.out‚ 8159.66015625 disk.sda1.out‚0 disk.dm-1.out‚ 68.890625 disk.sda3.out‚ 68.890625 network.sent‚954512 disk.nvme0n1.in‚ 2153.65625 cpu‚25.477567731576944 disk.rbd1.in‚ 0.00390625 disk.rbd0.out‚0 disk.rbd2.out‚ 4.3896484375 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 11.39453125( disk./.usagePercent‚38.6922634400269 disk.sda2.in‚0$ disk./.usageGB‚576.5426864624023£ÏQϔĂ(:‰ Â¥êÒÕ𩬠gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§2ŽªýÅ‚(:ò Â¥êÒ†¡ã¯ disk.rbd2.out‚ 4.52392578125 disk.drbd2165.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 2153.8671875 cpu‚25.475795476209527 proc.cpu.threads‚17$ disk./.usageGB‚576.5396347045898 disk.rbd1.out‚ 8287.67578125 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚959516 disk.sda.out‚ 71.328125 disk.sda2.in‚0 disk.dm-0.in‚ 2153.8671875 network.recv‚285624 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚7665.41064453125* proc.memory.percent‚17.345857620239258( disk./.usagePercent‚38.6920586339995% disk.nvme0n1.out‚7665.41064453125 disk.dm-1.out‚ 71.328125 disk.sda3.in‚0 disk.sda3.out‚ 71.328125" proc.memory.rssMB‚ 2841.9453125 disk.rbd1.in‚ 0.00390625" disk.dm-0.out‚7665.41064453125% proc.memory.availableMB‚ 11.390625 disk.nvme0n1.in‚ 2153.8671875 disk.sda2.out‚0$ memory_percent‚99.93047714233398 disk.rbd0.in‚0`éAܔƂ(:‰ Ñ¥êÒ¬Òç¬ gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª’6*Ç‚(:ú Ñ¥êÒ³Çù­ disk.sda3.in‚0 disk.rbd1.in‚ 0.00390625 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda3.out‚ 71.96484375 proc.cpu.threads‚17$ disk.nvme0n1.out‚7678.0615234375* disk./.usagePercent‚38.692065290195394 disk.sda2.out‚0 disk.rbd2.out‚ 4.52392578125! disk.dm-0.out‚7678.0615234375* proc.memory.percent‚17.345857620239258 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 2154.48828125# memory_percent‚99.9305248260498 disk.sda.out‚ 71.96484375 disk.dm-1.in‚0 disk.dm-1.out‚ 71.96484375" proc.memory.rssMB‚ 2841.9453125 network.recv‚289123 disk.rbd1.out‚ 8295.86328125 disk.dm-0.in‚ 2154.48828125 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 network.sent‚964551$ disk./.usageGB‚576.5397338867188 disk.sda.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 11.3828125 cpu‚25.474504819443833 disk.sda1.in‚0& disk.nvme0c0n1.out‚7678.0615234375! disk.nvme0n1.in‚ 2154.48828125.Ñ!gžÈ‚(:“ à¥êÒòÜò« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.087' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.81739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hù.böÉ‚(:ë à¥êÒàòš® disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 72.671875 disk.rbd1.in‚ 0.00390625* proc.memory.percent‚17.345857620239258 proc.cpu.threads‚17 network.sent‚969554 disk.rbd0.in‚0 disk.dm-0.in‚ 2155.4453125! disk.dm-0.out‚7686.9814453125 disk.rbd0.out‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚7686.9814453125 disk.sda2.in‚0$ disk./.usageGB‚576.5376472473145 disk.dm-1.out‚ 72.671875 disk.sda.in‚0 cpu‚25.47265069996833* disk./.usagePercent‚38.691925254074164 disk.sda2.out‚0" proc.memory.availableMB‚11.375 network.recv‚292340 disk.rbd2.out‚ 4.994140625 disk.sda1.in‚0 disk.sda.out‚ 72.671875" proc.memory.rssMB‚ 2841.9453125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 2155.4453125 disk.rbd1.out‚ 8518.75390625$ memory_percent‚99.93057250976562 disk.rbd2.in‚0.015625" disk.nvme0c0n1.in‚ 2155.4453125& disk.nvme0c0n1.out‚7686.9814453125Ä˵«0Ê+ _wandbruntime‚992  _runtime‚992Ì‹!©Â˼ _step‚45 & lr-AdamW/pg1‚0.0002739967064354332 & lr-AdamW/pg2‚0.0002739967064354332 trainer/global_step‚599 $ _timestamp‚1.784320738698769e+09 _runtime‚ 992.07336873oOûDÆÌÀ & lr-AdamW/pg2‚0.0002739967064354332 trainer/global_step‚599 $ _timestamp‚1.784320738698769e+09 _runtime‚ 992.07336873 _step‚45 & lr-AdamW/pg1‚0.0002739967064354332-¾ý«n0Í+ _wandbruntime‚992  _runtime‚992Ìì*n¦ ÎÐO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46731708215296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171276824470528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 559969268233 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171278196032512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 % _timestamp‚1.7843207386998448e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5059971 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170715895570432 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5059971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171202803538691 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1538917 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5059977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5174259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170641489670144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3234630 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170715921743872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 562274288640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171278196032512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171276824470528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1538922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170715895570432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 992.074683763 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721909138432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 562274288640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5164288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0—>r"Ú'ÏÔO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170715921743872 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 560928900096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47453617353728 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 721945773056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46732076306432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1538917 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5059977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5174259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170641489670144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3234630 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170715921743872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 562274288640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171278196032512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171276824470528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1538922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170715895570432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 992.074683763 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721909138432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 562274288640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5164288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46731708215296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171276824470528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 559969268233 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171278196032512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 % _timestamp‚1.7843207386998448e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5059971 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170715895570432 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5059971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171202803538691 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3239642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46809194622976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 trainer/global_step‚599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170993922473984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5182751 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171557121805312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171481608783225 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5182751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561853935616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5068183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5068177 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723127733248 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 560892734479 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171555750236160 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5068177 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47532285728768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5068183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170993896300544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 723091105792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1541465 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170993922473984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170919371448320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170919346020352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3239906 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171480238754831 _step‚47 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46809562714112 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843207400853336e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1541460 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171557121805312 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 561853935616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5172766 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 563199331328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 563199331328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5172766 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 563199331328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 563199331328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3239642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46809194622976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 trainer/global_step‚599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170993922473984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5182751 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171557121805312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171481608783225 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5182751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561853935616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5068183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5068177 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723127733248 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 560892734479 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171555750236160 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5068177 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47532285728768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QEDeviceStatsMonitor.on_train_vÆ­!batch_end/allocation.large_pool.allocated‚5068183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170993896300544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 723091105792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1541465 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170993922473984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170919371448320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170919346020352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3239906 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171480238754831 _step‚47 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46809562714112 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843207400853336e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1541460 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171557121805312 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 561853935616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5600011 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3499060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185223176906752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 607244830720 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5470277 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185224548468736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1832724 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184615958249472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51387219057152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50606028599296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 607244830720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1666067 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185224548468736 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5590040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184615958249472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5590040 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1666072 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51387623782912 _step‚50 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5600011 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184615932076032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 781227092480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 607550981655 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5470277 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184615932076032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184535553150976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008÷kûU åÕO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5590040 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1666072 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51387623782912 _step‚50 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5600011 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184615932076032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 781227092480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 607550981655 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5470277 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184615932076032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184535553150976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5600011 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3499060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185223176906752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 607244830720 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5470277 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185224548468736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1832724 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184615958249472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51387219057152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50606028599296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 607244830720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1666067 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185224548468736 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5590040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184615958249472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_tÙ^ô†rain_batch_start/segment.small_pool.current‚820 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5478477 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 609515261952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185503474241536 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1668610 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184893958979584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 782409773568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 608474448013 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 608169866240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1835464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚51 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3504074 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51466292871680 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 609515261952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5478477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 782373146112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185421934805133 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7843208185809977e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185503474241536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5598518 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50683883098112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚ 1071.95588675 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51465888153088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185420564776739 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184813434929152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1668615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184893958979584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 607129847587 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5478483 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1835719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 608169866240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184893932806144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5478477 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 609515261952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185503474241536 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1668610 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184893958979584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 782409773568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 608474448013 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 608169866240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1835464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚51 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3504074 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51466292871680 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 609515261952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5478477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 782373146112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185421934805133 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7843208185809977e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185503474241536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5598518 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50683883098112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚ 1071.95588675 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51465888153088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185420564776739 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184813434929152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16686153S1%2é- _wandbruntime‚1072  _runtime‚1072ËÅM!êý 1loss/SRqqt2/detection_loss‚2.0293246768666986 /loss/FBt2/detection_loss‚3.1483920056645465 &loss/total_loss‚22.677516885077694 epoch‚1 % _timestamp‚1.7843208186668036e+09 /loss/FRt1/assignment_loss‚2.901653081178665 /loss/FRt2/assignment_loss‚4.046707661255546 2loss/SRqqt2/assignment_loss‚1.5242789848297265 /loss/FRt2/detection_loss‚0.5979967870117564 _runtime‚1072.041202681 _step‚52 2loss/SRqqt1/assignment_loss‚1.5091953919045529 1loss/FBt1/assignment_loss‚0.06790503919639106 /loss/FRt1/detection_loss‚1.3654644985674822 1loss/FBt2/assignment_loss‚0.28894530356213116 1loss/SRqqt1/detection_loss‚0.7058136979941037 .loss/FBt1/detection_loss‚4.491839757046095 trainer/global_step‚649:„ ë 1loss/FBt2/assignment_loss‚0.28894530356213116 1loss/SRqqt1/detection_loss‚0.7058136979941037 .loss/FBt1/detection_loss‚4.491839757046095 trainer/global_step‚649 1loss/SRqqt2/detection_loss‚2.0293246768666986 /loss/FBt2/detection_loss‚3.1483920056645465 &loss/total_loss‚22.677516885077694 epoch‚1 % _timestamp‚1.7843208186668036e+09 /loss/FRt1/assignment_loss‚2.901653081178665 /loss/FRt2/assignment_loss‚4.046707661255546 2loss/SRqqt2/assignment_loss‚1.5242789848297265 /loss/FRt2/detection_loss‚0.5979967870117564 _runtime‚1072.041202681 _step‚52 2loss/SRqqt1/assignment_loss‚1.5091953919045529 1loss/FBt1/assignment_loss‚0.06790503919639106 /loss/FRt1/detection_loss‚1.36546449856748224>NL“ì‚(:ˆ º¦êÒŠÄ‚¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žè7í‚(:ù º¦êÒ¬Û—¯ disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 2307.59765625% disk.nvme0n1.out‚7750.12255859375 disk.sda1.in‚0 disk.dm-1.out‚ 80.7578125# proc.memory.availableMB‚10.9375$ disk./.usageGB‚576.5351104736328 disk.sda3.out‚ 80.7578125 disk.rbd1.in‚ 0.00390625" disk.dm-0.out‚7750.12255859375$ memory_percent‚99.93324279785156* proc.memory.percent‚17.345857620239258 network.recv‚313265 disk.sda.out‚ 80.7578125 disk.sda2.out‚0 network.sent‚ 1.058122e+06 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2307.59765625 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 5.6064453125" proc.memory.rssMB‚ 2841.9453125( disk./.usagePercent‚38.6917550090639' disk.nvme0c0n1.out‚7750.12255859375 disk.sda.in‚0 disk.dm-0.in‚ 2307.59765625 cpu‚25.4641963254074 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 9041.19140625G¸Ÿî‚(:” ɦêÒû¡ë« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.073800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b²kß ï‚(:€ ɦêÒÕȆ¯ disk.sda2.in‚0 disk.rbd1.out‚ 9062.58203125" proc.memory.rssMB‚ 2841.9453125* proc.memory.percent‚17.345857620239258# disk./.usageGB‚581.140697479248 disk.sda2.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚39.000839817355626 disk.dm-0.in‚ 2307.59765625# disk.dm-0.out‚11656.72509765625 disk.sda3.out‚ 83.546875 cpu‚25.463120213704375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.out‚ 83.546875 disk.sda1.in‚0& disk.nvme0n1.out‚11656.72509765625 disk.sda.in‚0 network.sent‚ 1.120416e+06 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 5.6064453125 disk.dm-1.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 2307.59765625$ memory_percent‚99.93269443511963 network.recv‚318339 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2307.59765625( disk.nvme0c0n1.out‚11656.72509765625 disk.sda.out‚ 83.546875 disk.rbd1.in‚ 0.00390625' proc.memory.availableMB‚ 11.02734375 disk.sda1.out‚0â–4”ð‚(:‰ ئêÒˆóï« gpu.0.gpu‚78.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Ž¹ñ‚(:ƒ ئêÒâû€¯ disk.drbd2165.in‚0( disk.nvme0c0n1.out‚12495.58642578125& disk.nvme0n1.out‚12495.58642578125 disk.rbd1.out‚ 9204.08203125" proc.memory.rssMB‚ 2841.9453125 cpu‚25.462287788083017 disk.sda1.out‚0* disk./.usagePercent‚39.000451709933714 disk.sda3.in‚0 disk.rbd1.in‚ 0.00390625' proc.memory.availableMB‚ 10.76953125 network.recv‚321658 disk.sda1.in‚0 disk.nvme0n1.in‚ 2308.4296875 disk.sda.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 5.66943359375 disk.dm-1.out‚ 87.01171875 disk.sda3.out‚ 87.01171875 network.sent‚ 1.12547e+06$ disk./.usageGB‚581.1349143981934 disk.rbd2.in‚0.015625 disk.dm-1.in‚0* proc.memory.percent‚17.345857620239258 disk.rbd0.out‚0 disk.dm-0.in‚ 2308.4296875" disk.nvme0c0n1.in‚ 2308.4296875 disk.sda.out‚ 87.01171875 disk.sda2.in‚0# memory_percent‚99.9342679977417# disk.dm-0.out‚12495.58642578125 disk.drbd2165.out‚0 disk.rbd0.in‚0‘Í&Á”ò‚(:‰ ç¦êÒ˜ðë« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FöFH ó‚(: ç¦êÒà«ž® disk.sda1.out‚0( disk.nvme0c0n1.out‚12508.45654296875 disk.rbd1.out‚ 9233.92578125 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 2309.38671875 cpu‚25.46085825909607 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.00390625# disk.dm-0.out‚12508.45654296875 disk.sda3.out‚ 88.2265625 network.sent‚ 1.129587e+06 network.recv‚324196& disk.nvme0n1.out‚12508.45654296875 disk.sda.out‚ 88.2265625& proc.memory.availableMB‚ 11.0078125" proc.memory.rssMB‚ 2841.9453125* proc.memory.percent‚17.345857620239258# disk./.usageGB‚581.134895324707 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0) disk./.usagePercent‚39.00045042989604 disk.rbd2.out‚ 5.66943359375 disk.dm-1.out‚ 88.2265625# disk.nvme0c0n1.in‚ 2309.38671875 disk.rbd0.in‚0$ memory_percent‚99.93281364440918! disk.nvme0n1.in‚ 2309.38671875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0} –’ô‚(:‡ ö¦êҀ膭 gpu.0.gpu‚94.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ú zÿõ‚(:ô ö¦êÒƒ®ž®* disk./.usagePercent‚39.000448125828235! disk.nvme0n1.in‚ 2379.16015625 disk.rbd0.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 2379.16015625 disk.sda.in‚0 disk.sda.out‚ 88.7421875 disk.dm-0.out‚ 12511.515625 disk.sda3.in‚0! disk.nvme0n1.out‚ 12511.515625 disk.dm-0.in‚ 2379.16015625 disk.sda1.out‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2841.9453125 proc.cpu.threads‚17 disk.sda2.out‚0$ disk./.usageGB‚581.1348609924316 disk.rbd1.out‚ 9380.01953125 network.recv‚327497 disk.rbd2.in‚0.015625 network.sent‚ 1.135141e+06# disk.nvme0c0n1.out‚ 12511.515625 disk.sda2.in‚0 disk.rbd1.in‚ 0.00390625* proc.memory.percent‚17.345857620239258 disk.dm-1.out‚ 88.7421875' proc.memory.availableMB‚ 10.99609375 cpu‚25.460074540935363 disk.sda3.out‚ 88.7421875 disk.rbd0.in‚0 disk.rbd2.out‚ 5.732421875 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚99.93288516998291ìézå2ö- _wandbruntime‚1149  _runtime‚1149 ÞsøÇ÷Á _step‚53 ' lr-AdamW/pg1‚0.00027390906836397065 ' lr-AdamW/pg2‚0.00027390906836397065 trainer/global_step‚699 % _timestamp‚1.7843208957110093e+09 _runtime‚1149.0856847368/‹ˆËøÅ _runtime‚1149.085684736 _step‚53 ' lr-AdamW/pg1‚0.00027390906836397065 ' lr-AdamW/pg2‚0.00027390906836397065 trainer/global_step‚699 % _timestamp‚1.7843208957110093e+095 E÷^2ù- _wandbruntime‚1149  _runtime‚1149î’£¤úÑO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199169529342976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 653560761344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1969373 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 840708809216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_start/allocated_byteƒoò3"s.large_pool.freed‚198515968581632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1793222 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843208957122705e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 654906149888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 840674271744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚699 _runtime‚1149.087024546 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198515994755072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5880571 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 652443494449 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6015792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198429642059776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55321023255040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55321425883648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34537472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198515968581632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199169529342976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34537472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198515968581632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199169529342976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199169529342976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 653560761344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1969373 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 840708809216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198515968581632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1793222 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843208957122705e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 654906149888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 840674271744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚699 _runtime‚1149.087024546 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198515994755072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5880571 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚06;ý†2ü- _wandbruntime‚1150  _runtime‚1150ÿó‰á&ýÛM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55399695342080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198793995485184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199360890798647 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 655831192576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54557835390976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5888777 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6034255 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843208970972316e+09 _step‚55 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198793969311744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199449826677760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5888783 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 654485796864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198793995485184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1972383 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6034255 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1795765 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5888777 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 841896578560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198707549265920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6024270 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1972122 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54558203482112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3767882 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985Ä•G3¼þßM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198793995485184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199360890798647 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 655831192576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54557835390976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5888777 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6034255 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843208970972316e+09 _step‚55 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198793969311744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199449826677760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5888783 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 654485796864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198793995485184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1972383 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6034255 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1795765 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5888777 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 841896578560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198707549265920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6024270 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1972122 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54558203482112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3767882 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55399695342080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚119130477nLdÌ2ÿ- _wandbruntime‚1150  _runtime‚1150Òz«[€ü %loss/total_loss‚19.65112791507568 _runtime‚1150.557738219 _step‚56 /loss/FRt1/assignment_loss‚2.847982147044564 2loss/SRqqt1/assignment_loss‚1.2951693981985517 2loss/SRqqt2/assignment_loss‚1.4885635786779123 1loss/FBt2/assignment_loss‚0.12798592034901254 0loss/SRqqt2/detection_loss‚1.636969901394703 /loss/FBt2/detection_loss‚3.9528015426106076 epoch‚1 trainer/global_step‚699 /loss/FRt1/detection_loss‚1.3871114825358326 /loss/FRt2/detection_loss‚0.5844504497864824 /loss/FBt1/detection_loss‚1.5398729086552636 % _timestamp‚1.7843208971833963e+09 /loss/FRt2/assignment_loss‚3.955838464218893 1loss/FBt1/assignment_loss‚0.12212280235800695 1loss/SRqqt1/detection_loss‚0.7122593192458507ß«Êç€ 1loss/FBt2/assignment_loss‚0.12798592034901254 0loss/SRqqt2/detection_loss‚1.636969901394703 /loss/FBt2/detection_loss‚3.9528015426106076 epoch‚1 trainer/global_step‚699 /loss/FRt1/detection_loss‚1.3871114825358326 /loss/FRt2/detection_loss‚0.5844504497864824 /loss/FBt1/detection_loss‚1.5398729086552636 % _timestamp‚1.7843208971833963e+09 /loss/FRt2/assignment_loss‚3.955838464218893 1loss/FBt1/assignment_loss‚0.12212280235800695 1loss/SRqqt1/detection_loss‚0.7122593192458507 %loss/total_loss‚19.65112791507568 _runtime‚1150.557738219 _step‚56 /loss/FRt1/assignment_loss‚2.847982147044564 2loss/SRqqt1/assignment_loss‚1.2951693981985517 2loss/SRqqt2/assignment_loss‚1.48856357867791238Ôì ‚‚(:ú …§êÒ¾ò¥­ disk.drbd2165.in‚0 network.sent‚ 1.140265e+06& disk.nvme0c0n1.out‚17226.185546875 disk.dm-1.in‚0 network.recv‚331129 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.out‚ 89.32421875' proc.memory.availableMB‚ 10.93359375# disk./.usageGB‚581.134895324707 cpu‚25.45871813408218 disk.nvme0n1.in‚ 2379.9921875 disk.rbd1.out‚ 9411.69140625 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.345857620239258 disk.rbd1.in‚ 0.00390625 disk.dm-0.in‚ 2379.9921875 disk.dm-1.out‚ 89.32421875 disk.sda3.in‚0$ memory_percent‚99.93326663970947) disk./.usagePercent‚39.00045042989604 disk.sda.out‚ 89.32421875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2841.9453125 disk.rbd2.out‚ 5.732421875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 2379.9921875$ disk.nvme0n1.out‚17226.185546875 disk.sda.in‚0! disk.dm-0.out‚17226.185546875]°$"”ƒ‚(:‰ …§êÒˤè¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oùž„‚(:“ ”§êÒòŽà« gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.618' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.72359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜØž…‚(:û ”§êÒž”‹® disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚99.93224143981934 disk.nvme0n1.in‚ 2379.9921875 disk.sda.out‚ 89.671875 disk.rbd0.out‚0 disk.dm-0.in‚ 2379.9921875 disk.dm-1.in‚0 cpu‚25.457735892134803$ disk./.usageGB‚581.1355285644531 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 5.732421875 network.sent‚ 1.202509e+06 network.recv‚335589) disk./.usagePercent‚39.00049292714673 disk.sda2.out‚0 disk.sda.in‚0# disk.dm-0.out‚17229.07177734375 disk.drbd2165.in‚0& proc.memory.availableMB‚ 11.1015625( disk.nvme0c0n1.out‚17229.07177734375& disk.nvme0n1.out‚17229.07177734375 disk.rbd1.out‚ 9542.86328125 disk.sda1.in‚0 disk.dm-1.out‚ 89.671875 disk.sda3.out‚ 89.671875" disk.nvme0c0n1.in‚ 2379.9921875" proc.memory.rssMB‚ 2841.9453125* proc.memory.percent‚17.345857620239258 disk.rbd1.in‚ 0.00390625 u’†‚(:‡ £§êÒÏÔ“¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Ñ ‡‚(:€ £§êÒÅìÀ®$ memory_percent‚99.93324279785156# disk./.usageGB‚581.135856628418 disk.sda2.out‚0 disk.sda3.out‚ 89.97265625! disk.nvme0n1.in‚ 2380.94921875 disk.sda2.in‚0 disk.rbd2.out‚ 5.732421875# disk.nvme0c0n1.in‚ 2380.94921875 disk.rbd2.in‚0.015625# disk.dm-0.out‚17239.41455078125 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 89.97265625 proc.cpu.threads‚17 network.sent‚ 1.208069e+06 network.recv‚339053 disk.sda1.out‚0& disk.nvme0n1.out‚17239.41455078125 disk.rbd1.in‚ 0.00390625 disk.dm-1.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 2841.9453125) disk./.usagePercent‚39.00051494379467 disk.rbd1.out‚ 9630.12890625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 89.97265625* proc.memory.percent‚17.345857620239258 disk.dm-0.in‚ 2380.94921875( disk.nvme0c0n1.out‚17239.41455078125# proc.memory.availableMB‚10.9375 cpu‚25.456588319958744 disk.rbd0.in‚0 disk.rbd0.out‚0Ûb¡”ˆ‚(:‰ ²§êÒ·Šë« gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚326.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.2162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬þC› ‰‚(:þ ²§êÒÞÞ® disk.sda2.in‚0 disk.dm-0.in‚ 2469.55078125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚17247.3115234375 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 9716.91015625" proc.memory.rssMB‚ 2841.9453125 disk.sda3.out‚ 90.13671875 disk.sda.in‚0 disk.rbd2.in‚0.015625" disk.dm-0.out‚17247.3115234375 network.sent‚ 1.213083e+06 disk.rbd0.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 2469.55078125* proc.memory.percent‚17.345857620239258 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0$ memory_percent‚99.93476867675781 proc.cpu.threads‚17 network.recv‚342063$ disk./.usageGB‚581.1356086730957% disk.nvme0n1.out‚17247.3115234375 disk.sda.out‚ 90.13671875 disk.sda2.out‚0 disk.rbd2.out‚ 5.732421875 disk.dm-1.in‚0 disk.dm-1.out‚ 90.13671875) disk./.usagePercent‚39.00049830330495! disk.nvme0n1.in‚ 2469.55078125# proc.memory.availableMB‚10.6875 cpu‚25.455885171766454$` ”Š‚(:‰ Á§êÒ®µø« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å5h‹‚(:ý Á§êÒ”꟮ disk.drbd2165.in‚0% disk.nvme0n1.out‚17253.3056640625 disk.sda2.out‚0 network.sent‚ 1.217619e+06 disk.rbd0.in‚0$ disk./.usageGB‚581.1355133056641 disk.sda3.out‚ 90.58203125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 2469.55078125" disk.dm-0.out‚17253.3056640625$ memory_percent‚99.93476867675781* proc.memory.percent‚17.345857620239258 disk.dm-1.in‚0 disk.dm-1.out‚ 90.58203125 disk.sda.out‚ 90.58203125 disk.rbd2.in‚0.015625# proc.memory.availableMB‚10.6875 proc.cpu.threads‚17) disk./.usagePercent‚39.00049190311659 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 2469.55078125 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875" proc.memory.rssMB‚ 2841.9453125 network.recv‚345752 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2469.55078125 disk.rbd1.out‚ 9785.35546875 disk.rbd2.out‚ 5.732421875 cpu‚25.45462523951362 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚17253.3056640625þ)¤2Œ- _wandbruntime‚1227  _runtime‚1227óBÁWľ & lr-AdamW/pg1‚0.0002738520608584074 & lr-AdamW/pg2‚0.0002738520608584074 trainer/global_step‚749 $ _timestamp‚1.784320974620769e+09 _runtime‚1227.995382389 _step‚57n~@YÈŽÂ _step‚57 & lr-AdamW/pg1‚0.0002738520608584074 & lr-AdamW/pg2‚0.0002738520608584074 trainer/global_step‚749 $ _timestamp‚1.784320974620769e+09 _runtime‚1227.9953823899õ‰°î2- _wandbruntime‚1227  _runtime‚1227<‹¼ª×'ÑO bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 900109706240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4027497 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6290871 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 900073071616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212323756396544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6290877 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6451515 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58355037458432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213115881779200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6290877 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59254742438912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 699876691968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚749 % _timestamp‚1.7843209746218326e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213115881779200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 698680607557 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 701222080512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6290871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59255147164672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212416005087232 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 701222080512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚749 % _timestamp‚1.7843209746218326e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213115881779200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YHDeviceStatsMonitor.on_train_batch_starßð¤ t/requested_bytes.small_pool.freed‚ 698680607557 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 701222080512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6290871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59255147164672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212416005087232 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 701222080512 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213115881779200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314:jÇö2’- _wandbruntime‚1229  _runtime‚1229Àã,á&“ÛM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 702147123200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58432155774976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4032485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6299083 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6450022 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213394807544832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843209760078623e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212694031990784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212694005817344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901252241408 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 700801727488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2109575 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚59 trainer/global_step‚749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59333408016384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922910 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 702147123200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922915 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58432523866112 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 699604073803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6460007 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212601638174720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213302586848949 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 700948674229 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213396179113984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6299083 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚1229.382788588 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6299083 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚1229.382788588 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4032485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6299083 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6450022 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213394807544832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843209760078623e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212694031990784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212694005817344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901252241408 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 700801727488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2109575 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚59 trainer/global_step‚749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59333408016384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922910 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 702147123200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922915 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58432523866112 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 699604073803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6460007 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212601638174720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213302586848949 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 700948674229;‘iì2•- _wandbruntime‚1229  _runtime‚1229ÕÝ4îü–ö /loss/FRt2/assignment_loss‚3.842755504677123 +loss/FBt1/detection_loss‚4.933166048539 /loss/FRt1/assignment_loss‚2.641178380937741 0loss/FBt2/assignment_loss‚0.2522718098032503 /loss/FBt2/detection_loss‚1.8408787067119892 &loss/total_loss‚20.759806096251424 % _timestamp‚1.7843209760939472e+09 1loss/SRqqt1/assignment_loss‚1.162740165359923 1loss/FBt1/assignment_loss‚0.06964922219173347 /loss/FRt1/detection_loss‚1.3981988928229598 /loss/FRt2/detection_loss‚0.6174820477726168 _runtime‚1229.468306877 _step‚60 1loss/SRqqt2/assignment_loss‚1.506528292056613 1loss/SRqqt1/detection_loss‚0.6622362733503288 0loss/SRqqt2/detection_loss‚1.832720752028147 epoch‚1 trainer/global_step‚749‘éE«—ú /loss/FRt1/assignment_loss‚2.641178380937741 0loss/FBt2/assignment_loss‚0.2522718098032503 /loss/FBt2/detection_loss‚1.8408787067119892 &loss/total_loss‚20.759806096251424 % _timestamp‚1.7843209760939472e+09 1loss/SRqqt1/assignment_loss‚1.162740165359923 1loss/FBt1/assignment_loss‚0.06964922219173347 /loss/FRt1/detection_loss‚1.3981988928229598 /loss/FRt2/detection_loss‚0.6174820477726168 _runtime‚1229.468306877 _step‚60 1loss/SRqqt2/assignment_loss‚1.506528292056613 1loss/SRqqt1/detection_loss‚0.6622362733503288 0loss/SRqqt2/detection_loss‚1.832720752028147 epoch‚1 trainer/global_step‚749 /loss/FRt2/assignment_loss‚3.842755504677123 +loss/FBt1/detection_loss‚4.933166048539<:%Ϥ‘˜‚(:† ЧêÒ“šž­ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~cˆƒ™‚(:ƒ ЧêÒûñÒ®* proc.memory.percent‚17.345857620239258 network.recv‚348876* disk./.usagePercent‚38.825858923709916 disk.sda1.in‚0 disk.sda3.out‚ 90.75390625# disk.dm-0.out‚17260.83642578125" disk.nvme0c0n1.in‚ 2470.5078125' proc.memory.availableMB‚ 10.45703125 disk.sda.out‚ 90.75390625 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 9867.71484375 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 90.75390625 disk.sda3.in‚0 cpu‚25.453756363002338 disk.rbd2.out‚ 5.732421875 disk.drbd2165.in‚0 network.sent‚ 1.223168e+06 disk.sda2.out‚0 disk.rbd2.in‚0.015625 proc.cpu.threads‚17$ disk./.usageGB‚578.5333557128906 disk.nvme0n1.in‚ 2470.5078125( disk.nvme0c0n1.out‚17260.83642578125& disk.nvme0n1.out‚17260.83642578125 disk.dm-0.in‚ 2470.5078125 disk.sda1.out‚0$ memory_percent‚99.93617534637451 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2841.9453125ïgöŸš‚(:” ߧêÒúÓ‰­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ» <›‚(:ü ߧêÒƒñÈ® network.recv‚353016* proc.memory.percent‚17.345881462097168 disk.sda.out‚ 91.03515625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 91.03515625 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 network.sent‚ 1.285362e+06# disk.nvme0n1.out‚21975.46484375 disk.dm-0.in‚ 2471.33984375' proc.memory.availableMB‚ 10.61328125# proc.memory.rssMB‚ 2841.94921875 disk.sda3.out‚ 91.03515625 disk.sda.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 2471.33984375 disk.sda1.out‚0 cpu‚25.452535054781997 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2471.33984375% disk.nvme0c0n1.out‚21975.46484375 disk.rbd1.out‚ 9992.40234375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 5.732421875) disk./.usagePercent‚39.00051468778714$ disk./.usageGB‚581.1358528137207 disk.sda3.in‚0 disk.dm-0.out‚21975.46484375# memory_percent‚99.9352216720581(¡J*’œ‚(:‡ î§êÒÍ•¹« gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙîÁ¢ ‚(:‚ î§êÒ™­Š¯ disk.sda.out‚ 91.33984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 10.328125* proc.memory.percent‚17.345881462097168' disk.nvme0c0n1.out‚21978.9130859375 network.sent‚ 1.290428e+06 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.451899665294324 disk.drbd2165.in‚0% disk.nvme0n1.out‚21978.9130859375 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚10052.07421875 disk.rbd2.out‚ 5.732421875) disk./.usagePercent‚39.00051545580974 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 2580.40234375# proc.memory.rssMB‚ 2841.94921875 network.recv‚356175 disk.sda1.out‚0 disk.dm-1.out‚ 91.33984375" disk.dm-0.out‚21978.9130859375 proc.cpu.threads‚17$ disk./.usageGB‚581.1358642578125# disk.nvme0c0n1.in‚ 2580.40234375! disk.nvme0n1.in‚ 2580.40234375 disk.rbd2.in‚0.015625$ memory_percent‚99.93696212768555 disk.sda3.out‚ 91.33984375:6d¿”ž‚(:‰ ý§êÒÌ¢š¬ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q y‡Ÿ‚(:û ý§êÒòŠµ® disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 91.4609375) disk./.usagePercent‚39.00051238371933$ disk./.usageGB‚581.1358184814453 disk.sda3.in‚0 disk.rbd2.out‚ 5.732421875# disk.dm-0.out‚21988.75439453125$ memory_percent‚99.93562698364258% proc.memory.availableMB‚ 10.546875 network.recv‚359888( disk.nvme0c0n1.out‚21988.75439453125 disk.sda.out‚ 91.4609375 disk.nvme0n1.in‚ 2581.359375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 2581.359375 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚10178.65234375 disk.dm-1.in‚0 disk.rbd2.in‚0.015625 disk.rbd0.out‚0 disk.dm-0.in‚ 2581.359375* proc.memory.percent‚17.345881462097168 proc.cpu.threads‚17 network.sent‚ 1.296189e+06 disk.rbd0.in‚0# proc.memory.rssMB‚ 2841.94921875 cpu‚25.45075069437266 disk.sda3.out‚ 91.4609375& disk.nvme0n1.out‚21988.75439453125z˜íc” ‚(:‰ Œ¨êÒ±žÈ­ gpu.0.gpu‚97.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š;Ui¡‚(:† Œ¨êҧ쨮$ memory_percent‚99.93712902069092) disk./.usagePercent‚39.00051571181727! disk.nvme0n1.in‚ 2582.19140625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 2582.19140625' proc.memory.availableMB‚ 10.30078125 network.recv‚362919 disk.rbd2.in‚0.015625 cpu‚25.450175881249155" disk.dm-0.out‚21997.0380859375 disk.sda3.in‚0 disk.sda3.out‚ 91.65234375 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚10226.98046875 disk.sda1.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2841.94921875% disk.nvme0n1.out‚21997.0380859375 disk.sda.out‚ 91.65234375 disk.rbd2.out‚ 6.19091796875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2582.19140625' disk.nvme0c0n1.out‚21997.0380859375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚581.1358680725098 disk.rbd0.out‚0 disk.dm-1.out‚ 91.65234375* proc.memory.percent‚17.345881462097168 network.sent‚ 1.301422e+06 disk.sda2.in‚0<jž¢‚(:“ ›¨êÒå²Ø« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çcdð£‚(:„ ›¨êÒÇÐû­ disk.dm-1.out‚ 91.98046875 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚581.1345634460449 disk.drbd2165.out‚0 disk.dm-1.in‚0' proVc.memory.availableMB‚ 10.00390625# proc.memory.rssMB‚ 2841.94921875 network.recv‚366425 disk.rbd0.out‚0 disk.dm-0.in‚ 2582.19140625" disk.dm-0.out‚22002.9033203125% disk.nvme0n1.out‚22002.9033203125 disk.drbd2165.in‚0* proc.memory.percent‚17.345881462097168 disk.sda.in‚0 disk.sda2.out‚0) disk./.usagePercent‚39.00042815724056# disk.nvme0c0n1.in‚ 2582.19140625 disk.rbd0.in‚0$ memory_percent‚99.93894100189209 cpu‚25.44905973374481 disk.sda2.in‚0 disk.rbd2.out‚ 6.19091796875 disk.sda.out‚ 91.98046875 network.sent‚ 1.30592e+06 disk.sda3.out‚ 91.98046875' disk.nvme0c0n1.out‚22002.9033203125! disk.nvme0n1.in‚ 2582.19140625 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.rbd1.out‚10404.60546875,@Ø/2¤- _wandbruntime‚1307  _runtime‚1307­(Y~Å¥¿ _step‚61 & lr-AdamW/pg1‚0.0002737862657333877 & lr-AdamW/pg2‚0.0002737862657333877 trainer/global_step‚799 % _timestamp‚1.7843210537290328e+09 _runtime‚1307.103645821H=±É¦Ã & lr-AdamW/pg1‚0.0002737862657333877 & lr-AdamW/pg2‚0.0002737862657333877 trainer/global_step‚799 % _timestamp‚1.7843210537290328e+09 _runtime‚1307.103645821 _step‚61=X7Ae2§- _wandbruntime‚1307  _runtime‚1307Èþ>{Ö'¨ÐO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2244429 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6877267 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62228989751296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚799 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227062234215424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63188483745280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226316067766272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959493993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 746262320979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4292215 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 746192622592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚62 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6867296 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6877267 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6701171 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63188888471040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227063605777408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226217819877376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1307.10486776 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6701177 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4291946 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6867296 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6701177 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747538011136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226962737598041 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747538011136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226316041592832 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 746192622592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2244693 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843210537301438e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 746262320979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4292215 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 746192622592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚62 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6867296 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6877267 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6701171 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63188888471040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227063605777408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226217819877376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1307.10486776 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6701177 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4291946 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6867296 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6701177 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747538011136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226962737598041 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747538011136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226316041592832 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 746192622592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2244693 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843210537301438e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227062234215424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63188483745280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226316067766272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959493993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627>j®-÷2ª- _wandbruntime‚1308  _runtime‚1308`€núá&«ÛM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2247448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2247186 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226495701655552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 747117658112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2050065 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4297513 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960676673024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 747185787337 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226495727083520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 960713300480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6875774 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227341159981056 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4297246 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 747117658112 _step‚63 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6709377 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843210551175082e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226594068496384 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 745841186911 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227342531550208 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2050060 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748463053824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227241542842463 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226594068496384 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6709383 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62306844250112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748463053824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63267152832000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0h\Bñæ¬ßM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2050065 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4297513 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960676673024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 747185787337 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226495727083520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 960713300480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6875774 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227341159981056 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4297246 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 747117658112 _step‚63 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6709377 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843210551175082e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226594068496384 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 745841186911 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227342531550208 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2050060 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748463053824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227241542842463 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226594068496384 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6709383 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62306844250112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748463053824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63267152832000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2247448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2247186 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226495701655552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 747117658112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712?‘oGí2­- _wandbruntime‚1308  _runtime‚13081ïoÿ®ù 0loss/FRt1/assignment_loss‚2.4953103932205605 .loss/FRt1/detection_loss‚1.361670660459018 /loss/FRt2/detection_loss‚0.5871275510738585 1loss/SRqqt1/detection_loss‚0.6900191448437232 _step‚64 2loss/SRqqt1/assignment_loss‚1.0540301376974608 1loss/FBt1/assignment_loss‚0.12572650919659362 % _timestamp‚1.7843210552023351e+09 %loss/total_loss‚18.23697546372457 -loss/FRt2/assignment_loss‚3.7646186604107 2loss/SRqqt2/assignment_loss‚1.3063527470446221 0loss/FBt2/assignment_loss‚0.3649430939068174 0loss/SRqqt2/detection_loss‚1.848365305018749 trainer/global_step‚799 /loss/FBt2/detection_loss‚2.7814783599701007 epoch‚1 _runtime‚1308.576738881 /loss/FBt1/detection_loss‚1.8573329008823645gêL¯ý 0loss/SRqqt2/detection_loss‚1.848365305018749 trainer/global_step‚799 /loss/FBt2/detection_loss‚2.7814783599701007 epoch‚1 _runtime‚1308.576738881 /loss/FBt1/detection_loss‚1.8573329008823645 0loss/FRt1/assignment_loss‚2.4953103932205605 .loss/FRt1/detection_loss‚1.361670660459018 /loss/FRt2/detection_loss‚0.5871275510738585 1loss/SRqqt1/detection_loss‚0.6900191448437232 _step‚64 2loss/SRqqt1/assignment_loss‚1.0540301376974608 1loss/FBt1/assignment_loss‚0.12572650919659362 % _timestamp‚1.7843210552023351e+09 %loss/total_loss‚18.23697546372457 -loss/FRt2/assignment_loss‚3.7646186604107 2loss/SRqqt2/assignment_loss‚1.3063527470446221 0loss/FBt2/assignment_loss‚0.3649430939068174@-îðX’°‚(:‡ ª¨êÒ‹ˆ¬ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²[š±‚(:ù ª¨êÒ¨ù×® disk.dm-0.in‚ 2761.0703125" disk.nvme0c0n1.in‚ 2761.0703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 92.30859375 disk.sda.in‚0$ memory_percent‚99.97251033782959 cpu‚25.47097754577716 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2761.0703125$ disk.nvme0n1.out‚22010.650390625$ disk./.usageGB‚581.1349258422852 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 92.30859375 disk.rbd1.out‚10411.48046875& proc.memory.availableMB‚ 4.50390625 disk.rbd2.out‚6.234375 disk.dm-1.in‚0 disk.sda.out‚ 92.30859375 network.recv‚369809! disk.dm-0.out‚22010.650390625 disk.drbd2165.out‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚22010.650390625" proc.memory.rssMB‚ 2851.1953125* proc.memory.percent‚17.402315139770508 network.sent‚ 1.311528e+06* disk./.usagePercent‚39.000452477956316 disk.sda2.out‚0 disk.rbd2.in‚0.015625¬-°ù²‚(:î ¹¨êÒ±¯´®) disk./.usagePercent‚39.00068083667686 disk.rbd0.in‚0' disk.nvme0c0n1.out‚25449.9716796875 disk.nvme0n1.in‚ 2761.0703125 network.recv‚374430 disk.dm-1.in‚0$ disk./.usageGB‚581.1383285522461 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 2761.0703125% disk.nvme0n1.out‚25449.9716796875 disk.dm-0.in‚ 2761.0703125" memory_percent‚99.969482421875 proc.memory.availableMB‚5 cpu‚25.497600243317766 proc.cpu.threads‚17 network.sent‚ 1.373729e+06 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚10563.30859375 disk.rbd2.in‚0.015625 disk.rbd2.out‚6.234375" disk.dm-0.out‚25419.9716796875 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2865.71875 disk.dm-1.out‚ 92.53515625 disk.sda.out‚ 92.53515625 disk.sda1.out‚0 disk.sda3.out‚ 92.53515625 disk.sda.in‚0) proc.memory.percent‚17.49095916748047™¥¥ž³‚(:“ ¹¨êÒ÷Ï”® gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚326.905' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.38099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑZ”Ã2´- _wandbruntime‚1335  _runtime‚13356[¶¯.>µ¨| :"Purity/1FRt*SRqqt1FBt/event_purity‚0.26538535952568054 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.02958100102841854 +particle/specificity‚0.8106091022491455 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2555791437625885 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06327156722545624 /particle/accuracy_2_of_2‚0.9995874762535095 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48723462224006653 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6315544843673706 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.21060030162334442 particle/accuracy_1_of_1‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2265625 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005501925945281982 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.019999999552965164 (particle/f_score‚0.47652897238731384 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.0560748465359211 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9934817552566528 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09950906783342361 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.04912617802619934 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.1980786770582199 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_3_of_5‚1 _runtime‚1335.228153034 _step‚65 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11569781601428986 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6607130765914917 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8611111044883728 (particle/accuracy‚0.7502131462097168 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2770785987377167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6541510224342346 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1106279045343399 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.09977570176124573 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965938329696655 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1866084486246109 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32706525921821594 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09610778093338013 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.19667808711528778 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011260582134127617 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9954979419708252 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33052459359169006 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.012580297887325287 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5293946862220764 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.21144907176494598 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909323453903198 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.048257749527692795 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.17802345752716064 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2864909768104553 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20300985872745514 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09869255125522614 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11010846495628357 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.053862929344177246 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.17125990986824036 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5230530500411987 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006855699233710766 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9798184633255005 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.25924089550971985 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957101941108704 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_1_of_1‚0.11885523796081543 particle/accuracy_1_of_3‚1 /particle/accuracy_2_of_3‚0.9990074634552002 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9842378497123718 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18077528476715088 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09610778093338013 +jet/accuracy_2_of_4‚0.19622690975666046 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.0560748465359211 jet/accuracy_4_of_4‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.21689344942569733 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.011018907651305199 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9939531683921814 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.25745177268981934 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.28187882900238037 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2075551301240921 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11590106040239334 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.22721776366233826 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04479174688458443 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0036114805843681097 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9912140965461731 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.041824545711278915 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3213360011577606 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.10879629850387573 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0833333358168602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7590914964675903 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.06208076328039169 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13878443837165833 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02511468715965748 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.5677707195281982 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895197749137878 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.17125990986824036 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29410499334335327 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9787405729293823 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.20205695927143097 )loss/val_total_loss‚46.12852738084872 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9967276453971863 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2817172706127167 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.18937909603118896 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9969338178634644 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 jet/accuracy_2_of_5‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024815292563289404 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.27390241622924805 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.12833333015441895 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9684758186340332 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11159838736057281 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.28087329864501953 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18898668885231018 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12412452697753906 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.20091499388217926 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.18516305088996887 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 +jet/accuracy_2_of_2‚0.07602901011705399 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2135416716337204 particle/accuracy_5_of_5‚1 % _timestamp‚1.7843210818528142e+09 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7100411653518677 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2617899179458618 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922454357147217 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.16214244067668915 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.44631922245025635 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.25463324785232544 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3490232825279236 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9926664233207703 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.21247518062591553 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9750000238418579 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.25839000940322876 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9954979419708252 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_4_of_5‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11779092252254486 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18703705072402954 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18286527693271637 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5831171870231628 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12444871664047241 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.025931235402822495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06281713396310806 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2075551301240921 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.8996604084968567 /particle/accuracy_4_of_4‚0.6520439982414246 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0625 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5496903657913208 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2430555522441864 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19096915423870087 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.166273832321167 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.15402433276176453 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26204055547714233 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.1785058081150055 +jet/accuracy_1_of_2‚0.46177801489830017 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8611111044883728 particle/accuracy_4_of_5‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8270031809806824 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.23484504222869873 /particle/accuracy_3_of_3‚0.9904176592826843 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.055168647319078445 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9123085737228394 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13802745938301086 *jet/accuracy_1_of_3‚0.6301217079162598 ,jet/accuracy_3_of_4‚0.016556451097130775 trainer/global_step‚799 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856790900230408 *jet/accuracy_2_of_3‚0.2071237564086914 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.15612822771072388 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951764345169067 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33799463510513306 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27537161111831665 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9908007383346558 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07021669298410416 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2593291699886322 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8312070965766907 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13103854656219482 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0044351969845592976’UÃGó ¶¬| =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.17802345752716064 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2864909768104553 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20300985872745514 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09869255125522614 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11010846495628357 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.053862929344177246 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.17125990986824036 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5230530500411987 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006855699233710766 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9798184633255005 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.25924089550971985 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957101941108704 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_1_of_1‚0.11885523796081543 particle/accuracy_1_of_3‚1 /particle/accuracy_2_of_3‚0.9990074634552002 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9842378497123718 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18077528476715088 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09610778093338013 +jet/accuracy_2_of_4‚0.19622690975666046 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.0560748465359211 jet/accuracy_4_of_4‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.21689344942569733 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.011018907651305199 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9939531683921814 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.25745177268981934 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.28187882900238037 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2075551301240921 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11590106040239334 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.22721776366233826 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04479174688458443 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0036114805843681097 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9912140965461731 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.041824545711278915 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3213360011577606 8 Purity/ÿi†¢?12FRt1SRqqt1FBt/FRt_purity‚0.10879629850387573 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0833333358168602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7590914964675903 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.06208076328039169 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13878443837165833 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02511468715965748 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.5677707195281982 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895197749137878 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.17125990986824036 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29410499334335327 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9787405729293823 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.20205695927143097 )loss/val_total_loss‚46.12852738084872 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9967276453971863 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2817172706127167 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.18937909603118896 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9969338178634644 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 jet/accuracy_2_of_5‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024815292563289404 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.27390241622924805 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.12833333015441895 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9684758186340332 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11159838736057281 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.28087329864501953 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18898668885231018 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12412452697753906 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.20091499388217926 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.18516305088996887 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 +jet/accuracy_2_of_2‚0.07602901011705399 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2135416716337204 particle/accuracy_5_of_5‚1 % _timestamp‚1.7843210818528142e+09 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7100411653518677 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2617899179458618 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922454357147217 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.16214244067668915 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.44631922245025635 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.25463324785232544 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3490232825279236 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9926664233207703 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.21247518062591553 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9750000238418579 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.25839000940322876 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9954979419708252 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_4_of_5‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11779092252254486 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18703705072402954 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18286527693271637 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5831171870231628 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12444871664047241 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.025931235402822495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06281713396310806 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2075551301240921 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.8996604084968567 /particle/accuracy_4_of_4‚0.6520439982414246 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0625 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5496903657913208 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2430555522441864 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19096915423870087 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.166273832321167 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.15402433276176453 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26204055547714233 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.1785058081150055 +jet/accuracy_1_of_2‚0.46177801489830017 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8611111044883728 particle/accuracy_4_of_5‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8270031809806824 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.23484504222869873 /particle/accuracy_3_of_3‚0.9904176592826843 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.055168647319078445 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9123085737228394 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13802745938301086 *jet/accuracy_1_of_3‚0.6301217079162598 ,jet/accuracy_3_of_4‚0.016556451097130775 trainer/global_step‚799 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856790900230408 *jet/accuracy_2_of_3‚0.2071237564086914 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.15612822771072388 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951764345169067 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33799463510513306 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27537161111831665 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9908007383346558 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07021669298410416 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2593291699886322 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8312070965766907 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13103854656219482 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0044351969845592976 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.26538535952568054 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.02958100102841854 +particle/specificity‚0.8106091022491455 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2555791437625885 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06327156722545624 /particle/accuracy_2_of_2‚0.9995874762535095 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48723462224006653 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6315544843673706 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.21060030162334442 particle/accuracy_1_of_1‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2265625 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005501925945281982 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.019999999552965164 (particle/f_score‚0.47652897238731384 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.0560748465359211 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9934817552566528 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09950906783342361 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.04912617802619934 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.1980786770582199 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_3_of_5‚1 _runtime‚1335.228153034 _step‚65 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11569781601428986 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6607130765914917 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8611111044883728 (particle/accuracy‚0.7502131462097168 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2770785987377167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6541510224342346 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1106279045343399 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.09977570176124573 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965938329696655 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1866084486246109 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32706525921821594 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09610778093338013 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.19667808711528778 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011260582134127617 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9954979419708252 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33052459359169006 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.012580297887325287 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5293946862220764 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.21144907176494598 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909323453903198 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.048257749527692795 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0A8w#¯”·‚(:‰ ȨêÒ…¨ø« gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w€D°¸‚(:û ȨêÒØž¶¯ disk.dm-0.in‚ 2761.90234375 network.sent‚ 1.378775e+06 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 6.29736328125 proc.memory.rssMB‚ 2865.78125! disk.dm-0.out‚28808.064453125# disk.nvme0c0n1.in‚ 2761.90234375 disk.drbd2165.in‚0 disk.dm-1.out‚ 92.72265625! disk.nvme0n1.in‚ 2761.90234375 disk.rbd1.in‚ 0.01171875$ memory_percent‚88.12713623046875$ disk./.usageGB‚582.0826988220215 disk.sda3.out‚ 92.72265625) proc.memory.percent‚17.49134063720703& disk.nvme0c0n1.out‚28808.064453125 network.recv‚377614 disk.sda2.out‚0 cpu‚25.482129525329565$ disk.nvme0n1.out‚28808.064453125 disk.sda.out‚ 92.72265625# proc.memory.availableMB‚1945.25 disk.rbd1.out‚10571.49609375 disk.rbd2.in‚0.015625) disk./.usagePercent‚39.06405831786092 disk.sda3.in‚0 disk.sda2.in‚0=RéÇ”¹‚(:‰ רêÒäŠÝ¬ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÉo º‚(:ÿ רêÒ‚ñï® disk.sda1.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 2762.859375 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 network.sent‚ 1.43721e+06# disk.dm-0.out‚28817.05029296875* disk./.usagePercent‚39.064066510102016 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚28817.05029296875 disk.sda.out‚ 93.31640625 disk.rbd1.out‚10812.58984375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 2762.859375& disk.nvme0n1.out‚28817.05029296875( proc.memory.availableMB‚ 1945.4296875 disk.dm-0.in‚ 2762.859375 disk.sda1.out‚0 disk.sda3.out‚ 93.31640625 disk.sda2.in‚0) proc.memory.percent‚17.49134063720703 cpu‚25.480904372279873 network.recv‚382138 disk.rbd2.in‚0.015625$ memory_percent‚88.12603950500488 proc.memory.rssMB‚ 2865.78125# disk./.usageGB‚582.082820892334 disk.rbd2.out‚ 6.29736328125 disk.dm-1.out‚ 93.31640625 disk.sda3.in‚0 disk.sda.in‚0U‚Ÿ»‚(:” æ¨êÒà‡…¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øsˆÑ ¼‚(:‚ æ¨êÒè‹¥® disk.sda1.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 2850.01953125 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.in‚0 disk.sda3.out‚ 93.6640625 disk.sda.out‚ 93.6640625 proc.cpu.threads‚17# disk.dm-0.out‚28820.30224609375 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 1945.1796875 cpu‚25.47970644988205 disk.rbd2.out‚ 6.3603515625 disk.sda2.in‚0 network.sent‚ 1.442366e+06 disk.rbd0.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚28820.30224609375$ memory_percent‚88.12756538391113 proc.memory.rssMB‚ 2865.78125) proc.memory.percent‚17.49134063720703 disk.rbd0.out‚0 disk.dm-1.out‚ 93.6640625$ disk./.usageGB‚582.0823249816895 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2850.01953125 disk.rbd1.out‚10955.74609375 network.recv‚385516* disk./.usagePercent‚39.064033229122565 disk.dm-0.in‚ 2850.01953125 disk.sda3.in‚0( disk.nvme0c0n1.out‚28820.30224609375 disk.sda.in‚0¤¹|”½‚(:‰ õ¨ê҈槬 gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûô(:‡ õ¨êÒ¼´Å® disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2851.10546875 proc.memory.rssMB‚ 2865.78125 disk.drbd2165.out‚0 disk.sda3.out‚ 93.91015625 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 network.sent‚ 1.446852e+06 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 2851.10546875 disk.sda2.out‚0 disk.rbd1.out‚11070.48046875 disk.rbd0.out‚0 disk.rbd2.out‚ 6.3603515625 disk.dm-0.in‚ 2851.10546875 disk.rbd0.in‚0# disk.dm-0.out‚28833.91259765625 disk.sda1.in‚0$ memory_percent‚88.13064098358154& disk.nvme0n1.out‚28833.91259765625 network.recv‚388857* disk./.usagePercent‚38.755660121776025 disk.dm-1.in‚0) proc.memory.availableMB‚ 1944.67578125 proc.cpu.threads‚17$ disk./.usageGB‚577.4873428344727 disk.dm-1.out‚ 93.91015625 disk.sda.out‚ 93.91015625 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd2.in‚0.015625) proc.memory.percent‚17.49134063720703( disk.nvme0c0n1.out‚28833.91259765625 cpu‚25.479207103327802$R Ì”¿‚(:‰ „©êÒœÇè« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žú~ÿÀ‚(:ô „©êÒÙå·®& disk.nvme0c0n1.out‚28847.830078125 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0.015625( proc.memory.availableMB‚ 1944.9140625 cpu‚25.478034449226207 network.recv‚392260 disk.nvme0c0n1.in‚ 2852.0625 disk.drbd2165.out‚0$ disk.nvme0n1.out‚28847.830078125! disk.dm-0.out‚28847.830078125 disk.sda.out‚ 96.23046875 disk.rbd0.out‚0 disk.rbd2.out‚ 6.3603515625 proc.cpu.threads‚17$ disk./.usageGB‚577.4843101501465 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.in‚0$ memory_percent‚88.12918663024902 disk.sda1.in‚0 disk.rbd1.out‚11078.66796875 disk.dm-0.in‚ 2852.0625 network.sent‚ 1.451926e+06 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 2852.0625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2865.78125) proc.memory.percent‚17.49134063720703* disk./.usagePercent‚38.755456595786306 disk.dm-1.out‚ 96.23046875 disk.sda3.out‚ 96.23046875Fà 2Á- _wandbruntime‚1419  _runtime‚1419# r7ľ & lr-AdamW/pg2‚0.0002737116872148095 trainer/global_step‚849 $ _timestamp‚1.784321166592844e+09 _runtime‚1419.967467909 _step‚66 & lr-AdamW/pg1‚0.0002737116872148095»¥óŽÈà& lr-AdamW/pg1‚0.0002737116872148095 & lr-AdamW/pg2‚0.0002737116872148095 trainer/global_step‚849 $ _timestamp‚1.784321166592844e+09 _runtime‚1419.967467909 _step‚66BÚy­Ò2Ä- _wandbruntime‚1419  _runtime‚1419y3œÙ'ÅÓO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7243477 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244044085706752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244044059533312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244044085706752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4598981 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243937817718784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244842410698240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7243483 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚244734801683991 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244842410698240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 798351164928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚67 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67178970200064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799696553984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7243483 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2183590 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7362002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799696553984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7352030 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4598755 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244843782260736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7243477 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2415170 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚1419.968746072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6h}‹ödÆ×O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244842410698240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7243483 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚244734801683991 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IÍyplit_bytes.all.freed‚67178569669120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798328565525 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244044059533312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244842410698240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 798351164928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚67 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67178970200064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799696553984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7243483 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2183590 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7362002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799696553984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7352030 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4598755 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244843782260736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7243477 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2415170 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚1419.968746072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7243477 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244044085706752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244044059533312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244044085706752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4598981 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243937817718784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0C[2Ç- _wandbruntime‚1421  _runtime‚1421q+V‹ã&ÈÝM MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7370494 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245122708033536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245122708033536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14612191 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2186128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799252031883 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2418117 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 800621596672 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4604034 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚68 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245121336463872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244322060263424 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 799276200448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244322060263424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244322086436864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244215724924928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244215699496960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2417906 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 797907431453 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66228908789760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245013606928413 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843211679871726e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7251683 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 799276200448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67257270542848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚1421.362015828 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7360508 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244322086436864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66229276880896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚849 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7251689 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7370494 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245014976956811 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7251683 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244215724924928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244215699496960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2417906 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 797907431453 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66228908789760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245013606928413 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843211679871726e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7251683 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 799276200448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67257270542848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚1421.362015828 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7360508 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244322086436864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66229276880896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚849 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7251689 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7370494 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245014976956811 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7251683 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2186128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799252031883 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2418117 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 800621596672 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4604034 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚68 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992DŒuK%2Ê- _wandbruntime‚1421  _runtime‚14213ˆÚöËü _step‚69 /loss/FRt2/assignment_loss‚3.413937816176489 1loss/FBt1/assignment_loss‚0.11103497961649873 1loss/FBt2/assignment_loss‚0.14705739135520307 &loss/total_loss‚16.689822624597234 % _timestamp‚1.7843211680700667e+09 0loss/SRqqt2/detection_loss‚1.588388607776599 /loss/FBt1/detection_loss‚1.4361161884148668 epoch‚2 trainer/global_step‚849 /loss/FRt1/assignment_loss‚2.576619658263749 2loss/SRqqt2/assignment_loss‚1.2473131980875443 /loss/FRt2/detection_loss‚0.6118385631856581 1loss/SRqqt1/detection_loss‚0.7339596565708657 .loss/FBt2/detection_loss‚2.391843690026191 _runtime‚1421.444421842 2loss/SRqqt1/assignment_loss‚0.9695883095698473 /loss/FRt1/detection_loss‚1.4621245655537232SóÃ÷Ì€ /loss/FRt2/assignment_loss‚3.413937816176489 1loss/FBt1/assignment_loss‚0.11103497961649873 1loss/FBt2/assignment_loss‚0.14705739135520307 &loss/total_loss‚16.689822624597234 % _timestamp‚1.7843211680700667e+09 0loss/SRqqt2/detection_loss‚1.588388607776599 /loss/FBt1/detection_loss‚1.4361161884148668 epoch‚2 trainer/global_step‚849 /loss/FRt1/assignment_loss‚2.576619658263749 2loss/SRqqt2/assignment_loss‚1.2473131980875443 /loss/FRt2/detection_loss‚0.6118385631856581 1loss/SRqqt1/detection_loss‚0.7339596565708657 .loss/FBt2/detection_loss‚2.391843690026191 _runtime‚1421.444421842 2loss/SRqqt1/assignment_loss‚0.9695883095698473 /loss/FRt1/detection_loss‚1.4621245655537232 _step‚69ECSה͂(:‰ “©êÒš¹ü¬ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚292.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iíü΂(:ñ “©êÒšðõ® disk.sda1.in‚0$ memory_percent‚88.12775611877441) proc.memory.percent‚17.49134063720703 network.recv‚395682 disk.nvme0n1.in‚ 2852.0625 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 2852.0625& disk.nvme0n1.out‚28857.18505859375 disk.sda2.out‚0 disk.rbd1.out‚11286.80859375 disk.rbd2.out‚ 6.3603515625 disk.drbd2165.out‚0 network.sent‚ 1.457476e+06# disk./.usageGB‚577.482723236084( disk.nvme0c0n1.out‚28857.18505859375 disk.sda1.out‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2865.78125 proc.cpu.threads‚17) disk./.usagePercent‚38.75535009665206 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 2852.0625 disk.dm-1.out‚ 97.390625 disk.sda.out‚ 97.390625( proc.memory.availableMB‚ 1945.1484375 cpu‚25.47720928529144# disk.dm-0.out‚28857.18505859375 disk.sda3.in‚0 disk.sda3.out‚ 97.390625 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875xÅZžŸÏ‚(:” ¢©êÒ̂Ϭ gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚299.991' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.998200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ŸÏüЂ(:ñ ¢©êÒ¥Òñ® disk.sda3.in‚0 disk.rbd2.out‚ 6.3603515625" disk.dm-0.out‚28864.7373046875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 2933.15625 disk.rbd1.out‚11421.30859375 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.75536468908151 disk.sda.out‚ 97.765625 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.dm-1.out‚ 97.765625' disk.nvme0c0n1.out‚28864.7373046875 disk.dm-1.in‚0 network.sent‚ 1.519975e+06 disk.sda2.out‚0 disk.drbd2165.out‚0" memory_percent‚88.130784034729) proc.memory.availableMB‚ 1944.65234375 network.recv‚400127$ disk./.usageGB‚577.4829406738281% disk.nvme0n1.out‚28864.7373046875 disk.rbd2.in‚0.015625) proc.memory.percent‚17.49134063720703 disk.nvme0c0n1.in‚ 2933.15625 disk.dm-0.in‚ 2933.15625 proc.memory.rssMB‚ 2865.78125 cpu‚25.47641974969202 disk.sda3.out‚ 97.765625 disk.sda.in‚0 disk.sda2.in‚0¦-ÒÑ‚(:” ±©êÒÿ‡÷« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚296.511' N’i)Ígpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.302200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡òï‚ Ò‚(: ±©êÒ‰¸¯ disk.dm-0.in‚ 2934.11328125 disk.dm-1.in‚0$ memory_percent‚88.12942504882812) disk./.usagePercent‚38.75537313733014# disk.dm-0.out‚28875.00341796875 disk.sda1.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2934.11328125 disk.sda2.out‚0$ proc.memory.availableMB‚1944.875 disk.sda1.in‚0 disk.sda3.out‚ 98.08203125 disk.sda.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd1.out‚11546.62109375) proc.memory.percent‚17.49134063720703 network.recv‚403445 disk.rbd0.out‚0$ disk./.usageGB‚577.4830665588379 disk.dm-1.out‚ 98.08203125( disk.nvme0c0n1.out‚28875.00341796875& disk.nvme0n1.out‚28875.00341796875 network.sent‚ 1.524979e+06 disk.sda.out‚ 98.08203125 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 2934.11328125 proc.memory.rssMB‚ 2865.78125 cpu‚25.475302783443837 proc.cpu.threads‚17 disk.rbd2.out‚ 6.36035156254Ǹm”Ó‚(:‰ À©êÒ„øÓ« gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÐôÔ‚(:„ À©êÒ¡¸ø­ network.sent‚ 1.530636e+06 network.recv‚406753) disk./.usagePercent‚38.75539259390274 disk.rbd0.in‚0# disk.dm-0.out‚28878.38720703125$ memory_percent‚87.62707710266113 proc.memory.rssMB‚ 2865.78125 disk.rbd2.out‚ 6.3603515625 disk.sda2.out‚0 disk.dm-0.in‚ 2934.11328125 disk.sda3.out‚ 98.34765625 disk.sda.in‚0 disk.rbd1.out‚11554.62109375) proc.memory.percent‚17.49134063720703 cpu‚25.47475268216753 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 2934.11328125( disk.nvme0c0n1.out‚28878.38720703125& disk.nvme0n1.out‚28878.38720703125 disk.sda.out‚ 98.34765625 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 98.34765625 disk.rbd1.in‚ 0.01171875$ disk./.usageGB‚577.4833564758301 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 2934.11328125( proc.memory.availableMB‚ 2027.1796875YÑÄŸÕ‚(:” Ï©êÒûúƒ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/<ÞçÖ‚(:Ü Ï©êÒêˆø­ disk.dm-1.out‚ 98.75390625 disk.nvme0n1.in‚ 2934.9453125 disk.nvme0n1.out‚28887 disk.sda.out‚ 98.75390625 disk.sda2.in‚0 disk.rbd1.out‚11699.44921875 network.sent‚ 1.534351e+06 disk.rbd0.out‚0 disk.rbd2.in‚0.015625) proc.memory.percent‚17.49134063720703 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚409502 disk.dm-0.out‚28887 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚94.05856132507324 disk.rbd2.out‚ 6.3603515625 disk.sda2.out‚0' proc.memory.availableMB‚ 973.4453125 disk.dm-0.in‚ 2934.9453125) disk./.usagePercent‚38.75535419277261 disk.nvme0c0n1.out‚28887 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875 cpu‚25.47367915039621$ disk./.usageGB‚577.4827842712402 disk.sda3.in‚0 disk.sda3.out‚ 98.75390625" disk.nvme0c0n1.in‚ 2934.9453125 proc.memory.rssMB‚ 2865.78125¼f«€“ׂ(:ˆ Þ©êÒÓ܉¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä©C² Ø‚(:€ Þ©êÒææÆ® disk.rbd1.in‚0.015625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 98.97265625& disk.nvme0n1.out‚28894.67236328125 cpu‚25.47276876630445* disk./.usagePercent‚38.755384913676714$ disk./.usageGB‚577.4832420349121 disk.sda2.in‚0 disk.rbd1.out‚ 11939.921875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2865.26953125 disk.sda2.out‚0 disk.dm-0.in‚ 3027.9609375( proc.memory.percent‚17.4882173538208 proc.cpu.threads‚17 network.recv‚413040 disk.rbd2.out‚ 6.3603515625 disk.drbd2165.out‚0 disk.sda.out‚ 98.97265625 network.sent‚ 1.539916e+06# disk.dm-0.out‚28894.67236328125 disk.drbd2165.in‚0 disk.sda3.out‚ 98.97265625" disk.nvme0c0n1.in‚ 3027.9609375( disk.nvme0c0n1.out‚28894.67236328125 disk.nvme0n1.in‚ 3027.9609375 disk.sda.in‚0$ memory_percent‚94.04823780059814( proc.memory.availableMB‚ 975.13671875]¸%2Ù- _wandbruntime‚1500  _runtime‚1500QgÚZÄÚ¾ & lr-AdamW/pg1‚0.0002736283300927118 & lr-AdamW/pg2‚0.0002736283300927118 trainer/global_step‚899 $ _timestamp‚1.784321246750401e+09 _runtime‚1500.125014714 _step‚70N“'2ÈÛ & lr-AdamW/pg1‚0.0002736283300927118 & lr-AdamW/pg2‚0.0002736283300927118 trainer/global_step‚899 $ _timestamp‚1.784321246750401e+09 _runtime‚1500.125014714 _step‚70FÁ!¾ú2Ü- _wandbruntime‚1500  _runtime‚1500–K>0Ù'ÝÓO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4864859 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846012484608 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71113725344768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70026110857216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843212467515318e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7787754 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257831906627584 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7653783 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257944096038912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2310735 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2553874 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258675127705899 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚71 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7653783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2310740 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 843221078315 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 844667095552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258788763134464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 844667095552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257944122212352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7777782 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258790134696960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71113322716672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2554119 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1087579950592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7653777 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0‚ÃÁžÝ'Þ×O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2310740 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 843221078315 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 844667095552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258788763134464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 844667095552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257944122212352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7777782 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258790134696960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71113322716672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2554119 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1087579950592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7653777 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4864859 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846012484608 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71113725344768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70026110857216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843212467515318e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7787754 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257831906627584 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7653783 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257944096038912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2310735 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2553874 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258675127705899 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚71 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7653783Ge]G¢2ß- _wandbruntime‚1501  _runtime‚15019,»óàÜM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258222096769024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2556620 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258953932950321 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7796246 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 845592131072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7661983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 846937527296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70103597264896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4869898 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 845489144991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7786260 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2313283 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7661983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258109788405760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7661989 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7786260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088819214336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2556852 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/N2¨±Êreserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258222122942464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71192416479232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259069060469760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258222122942464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258222096769024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70103229173760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258109813833728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259069060469760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 846937527296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258955302978719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4870135 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1088784684544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 845592131072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7796246 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2313278 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71192013858304‹$Ræ&áàM _step‚72 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259067688900096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259069060469760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 846937527296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258955302978719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4870135 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1088784684544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 845592131072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7796246 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2313278 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71192013858304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258222096769024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2556620 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258953932950321 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7796246 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 845592131072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7661983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 846937527296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70103597264896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4869898 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 845489144991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7786260 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2313283 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7661983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258109788405760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7661989 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7786260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088819214336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2556852 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258222122942464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71192416479232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259069060469760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258222122942464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258222096769024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70103229173760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258109813833728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 BÆúè‚(:ï ü©êÒÈõ­ disk.rbd1.in‚0.046875# disk.dm-0.out‚28908.24267578125 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 3038.171875 proc.memory.rssMB‚ 2865.78125( disk.nvme0c0n1.out‚28908.24267578125$ disk./.usageGB‚577.4842376708984 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3038.171875 disk.dm-1.in‚0 cpu‚25.47122576816209 disk.rbd0.out‚0 disk.nvme0n1.in‚ 3038.171875& disk.nvme0n1.out‚28908.24267578125' proc.memory.availableMB‚ 974.3984375) proc.memory.percent‚17.49134063720703 proc.cpu.threads‚17* disk./.usagePercent‚38.755451731643156 disk.sda.in‚0 network.recv‚420701 disk.sda2.in‚0 disk.rbd1.out‚ 12059.0625 disk.sda1.out‚0 disk.rbd2.out‚ 6.3603515625 disk.sda3.in‚0$ memory_percent‚94.05274391174316 network.sent‚ 1.607395e+06 disk.dm-1.out‚99.59375 disk.sda3.out‚99.59375 disk.sda.out‚99.59375 disk.rbd0.in‚0 disk.drbd2165.in‚04\³ˆ’é‚(:‡ ‹ªêÒùÃÞ¬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.072' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ΟÅ%ê‚(:ø ‹ªêÒ¿¿®) disk./.usagePercent‚38.75544891556027# disk.rbd1.out‚12155.08642578125 cpu‚25.470684621019533$ disk./.usageGB‚577.4841957092285 disk.rbd0.out‚0 disk.rbd2.out‚ 6.3603515625" memory_percent‚94.051194190979 network.recv‚424112 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.49134063720703! disk.nvme0c0n1.in‚ 3038.171875 disk.nvme0n1.in‚ 3038.171875 disk.sda2.out‚0( proc.memory.availableMB‚ 974.65234375 proc.memory.rssMB‚ 2865.78125 disk.drbd2165.in‚0 disk.dm-0.in‚ 3038.171875 disk.sda3.out‚ 99.890625 disk.sda.in‚0 disk.rbd1.in‚0.046875# disk.dm-0.out‚28912.62060546875 disk.drbd2165.out‚0 network.sent‚ 1.612982e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 99.890625( disk.nvme0c0n1.out‚28912.62060546875& disk.nvme0n1.out‚28912.62060546875 disk.sda.out‚ 99.890625 proc.cpu.threads‚17,˜Ù«Ÿë‚(:” šªêÒݨɭ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.144' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.428799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰žK0 ì‚(: šªêÒÜ£†¯ disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0* disk./.usagePercent‚38.755452243658226# disk./.usageGB‚577.484245300293 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 100.1953125# disk.nvme0c0n1.in‚ 3128.22265625& proc.memory.availableMB‚ 975.140625 network.sent‚ 1.617448e+06" disk.dm-0.out‚28920.5771484375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.out‚ 6.3603515625 disk.sda1.out‚0 disk.sda.in‚0 cpu‚25.4696814345196 disk.sda2.in‚0# disk.rbd1.out‚12163.21142578125 disk.drbd2165.in‚0% disk.nvme0n1.out‚28920.5771484375$ memory_percent‚94.04821395874023) proc.memory.percent‚17.48824119567871' disk.nvme0c0n1.out‚28920.5771484375! disk.nvme0n1.in‚ 3128.22265625" proc.memory.rssMB‚ 2865.2734375 disk.rbd1.in‚0.046875 disk.dm-0.in‚ 3128.22265625 disk.drbd2165.out‚0 disk.sda3.out‚ 100.1953125 disk.sda.out‚ 100.1953125 network.recv‚427156 disk.rbd0.in‚0 disk.sda1.in‚0즾å”í‚(:‰ ©ªêÒäÑÈ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚200.317' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.0634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÕõ±î‚(:ý ©ªêÒó…Û°$ disk./.usageGB‚577.4841918945312 disk.rbd0.out‚0% disk.nvme0n1.out‚28930.6591796875 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.622455e+06 disk.drbd2165.in‚0 disk.sda.out‚ 100.359375 cpu‚25.468972814803596 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda3.out‚ 100.359375 disk.rbd2.out‚ 6.3603515625 disk.sda3.in‚0 disk.nvme0n1.in‚ 3129.0546875 proc.memory.rssMB‚ 2865.78125 disk.rbd1.in‚0.046875 disk.dm-0.in‚ 3129.0546875" disk.nvme0c0n1.in‚ 3129.0546875$ memory_percent‚94.05133724212646( proc.memory.availableMB‚ 974.62890625) proc.memory.percent‚17.49134063720703 disk.dm-1.in‚0 disk.sda.in‚0) disk./.usagePercent‚38.75544865955274 disk.rbd0.in‚0 disk.rbd2.in‚0.015625" disk.dm-0.out‚28930.6591796875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚430709# disk.rbd1.out‚12271.49267578125 disk.dm-1.out‚ 100.359375' disk.nvme0c0n1.out‚28930.6591796875¼qPe2ï- _wandbruntime‚1579  _runtime‚1579c3AÆðÀ _step‚74 ' lr-AdamW/pg1‚0.00027353619972096655 ' lr-AdamW/pg2‚0.00027353619972096655 trainer/global_step‚949 $ _timestamp‚1.784321326351252e+09 _runtime‚1579.725846569Y‚ KÊñÄ _step‚74 ' lr-AdamW/pg1‚0.00027353619972096655 ' lr-AdamW/pg2‚0.00027353619972096655 trainer/global_step‚949 $ _timestamp‚1.784321326351252e+09 _runtime‚1579.725846569JCY£ƒ2ò- _wandbruntime‚1579  _runtime‚1579 6µ˜%óÓO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73900063150080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5130167 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2437885 _runtime‚1579.727058457 trainer/global_step‚949 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1147813120512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75048278898688 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8213506 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 889458191423 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8203534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2437890 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271726020964352 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8064083 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271844132544512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271844158717952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 892328415232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5130414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75047876270592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8064077 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8203534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271725995536384 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271844158717952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 892328415232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272736487133184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2692524 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1147847657472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8064077 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73900431241216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272615453727807 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271844132544512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8213506 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272736487133184 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8064083 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 890802791741 % _timestamp‚1.7843213263523426e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890983026176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272735115570688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272616823756093 UADeviceStatsMonitor.on_train_batch_start/allocatësñAed_bytes.all.freed‚272735115570688 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272735115570688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272616823756093 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272735115570688 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5130167 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2437885 _runtime‚1579.727058457 trainer/global_step‚949 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1147813120512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75048278898688 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8213506 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 889458191423 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8203534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2437890 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271726020964352 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8064083 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271844132544512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271844158717952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 892328415232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5130414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75047876270592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8064077 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8203534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271725995536384 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271844158717952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 892328415232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272736487133184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2692524 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1147847657472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8064077 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73900431241216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272615453727807 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271844132544512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8213506 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272736487133184 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8064083 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 890802791741 % _timestamp‚1.7843213263523426e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890983026176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 893253457920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2694992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272895629000627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73977549557760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272122133274624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273014041336320 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8212012 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272122159448064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 890381657669 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8221998 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75126974080000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272003902742528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843213277443757e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5135656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75126571459072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5135420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚76 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8072283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2695223 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8221998 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 891908061696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2440428 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 891726258099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273015412905984 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8072283 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8072289 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8072283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2695223 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8221998 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 891908061696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2440428 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 891726258099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273015412905984 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8072283 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8072289 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 893253457920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2694992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272895629000627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73977549557760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272122133274624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273014041336320 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8212012 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272122159448064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 890381657669 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8221998 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75126974080000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272003902742528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843213277443757e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5135656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75126571459072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5135420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚76L[¡¿2ø- _wandbruntime‚1581  _runtime‚1581øœIùü _runtime‚1581.202179833 _step‚77 &loss/total_loss‚14.644380548723976 epoch‚2 .loss/FRt1/assignment_loss‚2.09816851536872 0loss/FRt2/assignment_loss‚3.3489058189892926 1loss/FBt1/assignment_loss‚0.16510357368525388 0loss/FBt2/assignment_loss‚0.3523421850177093 /loss/FRt1/detection_loss‚1.4484293117718754 /loss/FBt1/detection_loss‚1.0280163664685065 1loss/SRqqt1/assignment_loss‚0.707411958364019 2loss/SRqqt2/assignment_loss‚1.1660855553142753 1loss/SRqqt1/detection_loss‚0.7894176950314605 1loss/SRqqt2/detection_loss‚1.3365585347935822 trainer/global_step‚949 % _timestamp‚1.7843213278278568e+09 /loss/FRt2/detection_loss‚0.6756618034068589 /loss/FBt2/detection_loss‚1.5282792305124218ÇQ¿Œú€ 1loss/SRqqt2/detection_loss‚1.3365585347935822 trainer/global_step‚949 % _timestamp‚1.7843213278278568e+09 /loss/FRt2/detection_loss‚0.6756618034068589 /loss/FBt2/detection_loss‚1.5282792305124218 _runtime‚1581.202179833 _step‚77 &loss/total_loss‚14.644380548723976 epoch‚2 .loss/FRt1/assignment_loss‚2.09816851536872 0loss/FRt2/assignment_loss‚3.3489058189892926 1loss/FBt1/assignment_loss‚0.16510357368525388 0loss/FBt2/assignment_loss‚0.3523421850177093 /loss/FRt1/detection_loss‚1.4484293117718754 /loss/FBt1/detection_loss‚1.0280163664685065 1loss/SRqqt1/assignment_loss‚0.707411958364019 2loss/SRqqt2/assignment_loss‚1.1660855553142753 1loss/SRqqt1/detection_loss‚0.7894176950314605M^a|¢mû‚(:‰ ¸ªêÒÝØ« gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚218.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚´¤ '0# gpu.0.uncorrectedMemoryErrors‚0cü‚(:„ ¸ªêÒÂ矮& proc.memory.availableMB‚ 974.609375 network.recv‚434076) disk./.usagePercent‚38.75545787582397 disk.sda1.out‚0) proc.memory.percent‚17.49134063720703$ disk./.usageGB‚577.4843292236328 disk.sda3.out‚ 100.55078125 disk.sda2.in‚0$ memory_percent‚94.05145645141602 disk.rbd2.out‚ 6.3603515625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17# disk.rbd1.out‚12418.93017578125 disk.rbd0.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 3129.0546875 disk.nvme0n1.in‚ 3129.0546875& disk.nvme0n1.out‚28934.46826171875 disk.rbd1.in‚0.046875 disk.dm-0.in‚ 3129.0546875# disk.dm-0.out‚28934.46826171875( disk.nvme0c0n1.out‚28934.46826171875 disk.sda.in‚0 proc.memory.rssMB‚ 2865.78125 disk.rbd0.in‚0 disk.sda3.in‚0 cpu‚25.468258619439514 network.sent‚ 1.627626e+06 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.out‚ 100.55078125 disk.sda.out‚ 100.55078125x«fžý‚(:“ ǪêÒúîÙ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.484' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.49679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kOþ‚(:„ ǪêÒ‚£°®* disk./.usagePercent‚38.755521365692466 disk.sda3.in‚0% disk.nvme0n1.out‚28950.8466796875 disk.sda.out‚ 102.7578125 proc.cpu.threads‚17 network.sent‚ 1.691157e+06 disk.drbd2165.in‚0 disk.dm-1.out‚ 102.7578125 disk.sda2.out‚0 proc.memory.rssMB‚ 2865.78125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3130.01171875 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3130.01171875$ memory_percent‚94.05410289764404) proc.memory.percent‚17.49134063720703 disk.rbd2.out‚ 6.3603515625 network.recv‚438758 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.in‚0.046875 cpu‚25.467472923921477( proc.memory.availableMB‚ 974.17578125$ disk./.usageGB‚577.4852752685547 disk.sda1.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚28950.8466796875 disk.dm-1.in‚0! disk.nvme0n1.in‚ 3130.01171875' disk.nvme0c0n1.out‚28950.8466796875 disk.sda3.out‚ 102.7578125# disk.rbd1.out‚12499.94580078125 disk.rbd0.out‚0Èóžÿ‚(:“ ÖªêÒ££ð¬ gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚340.354' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.07079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ð€‚(:‰ ÖªêÒö–ë® disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 6.4228515625 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 1.696185e+06* disk./.usagePercent‚38.755521109684935 disk.dm-0.in‚ 3203.69921875# disk.nvme0c0n1.in‚ 3203.69921875 disk.dm-1.out‚ 103.1484375& disk.nvme0n1.out‚28959.79150390625 disk.sda.in‚0 cpu‚25.466994870643575# disk.dm-0.out‚28959.79150390625 disk.dm-1.in‚0# disk.rbd1.out‚12516.07080078125 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda.out‚ 103.1484375 disk.sda2.in‚0$ memory_percent‚94.05572414398193( proc.memory.availableMB‚ 973.91015625" proc.memory.rssMB‚ 2865.2890625 disk.rbd1.in‚0.046875 network.recv‚441971 disk.sda1.out‚0 disk.sda3.out‚ 103.1484375 disk.sda2.out‚0) proc.memory.percent‚17.48833656311035$ disk./.usageGB‚577.4852714538574( disk.nvme0c0n1.out‚28959.79150390625! disk.nvme0n1.in‚ 3203.69921875†¿æGž‚(:“ åªêÒ†ÖÚ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚292.063' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.41259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vñ±‚‚(:‡ åªêÒþö°¯# disk.dm-0.out‚28966.58349609375 disk.sda3.out‚ 103.64453125 network.sent‚ 1.700719e+06 network.recv‚445427 disk.dm-0.in‚ 3203.69921875 disk.rbd1.in‚0.046875 disk.sda2.out‚0 proc.memory.rssMB‚ 2865.78125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 3203.69921875( disk.nvme0c0n1.out‚28966.58349609375# disk.rbd1.out‚12565.02392578125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 103.64453125 cpu‚25.466070429054344 disk.sda.out‚ 103.64453125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚94.05574798583984) proc.memory.percent‚17.49134063720703& disk.nvme0n1.out‚28966.58349609375 disk.sda.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 6.4228515625 disk.dm-1.in‚0$ disk./.usageGB‚577.4828567504883! disk.nvme0n1.in‚ 3203.69921875 disk.sda1.in‚0% proc.memory.availableMB‚ 973.90625 disk.sda3.in‚0* disk./.usagePercent‚38.755359056915765 disk.sda2.in‚0ƒiœƒ‚(:‘ ôªêÒ¡ÆØ« gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚329.8' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.96000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦)Z „‚(:ÿ ôªêÒüú¿®* disk./.usagePercent‚38.755357520870554 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3204.65625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 104.2265625# disk.dm-0.out‚28974.20263671875 disk.sda3.in‚0& disk.nvme0n1.out‚28974.20263671875 disk.sda.out‚ 104.2265625 network.recv‚448778 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚94.05581951141357( proc.memory.availableMB‚ 973.89453125 disk.rbd2.out‚ 6.88134765625 disk.sda3.out‚ 104.2265625( disk.nvme0c0n1.out‚28974.20263671875 proc.memory.rssMB‚ 2865.78125 network.sent‚ 1.705857e+06 disk.sda.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 3204.65625 disk.rbd1.in‚0.046875) proc.memory.percent‚17.49134063720703$ disk./.usageGB‚577.4828338623047 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 3204.65625# disk.rbd1.out‚12648.35205078125 cpu‚25.465506897184213 disk.sda2.in‚0 disk.drbd2165.out‚032…- _wandbruntime‚1659  _runtime‚1659f>¬Ìņ¿ & lr-AdamW/pg2‚0.0002734353020169356 trainer/global_step‚999 % _timestamp‚1.7843214058994792e+09 _runtime‚1659.274070957 _step‚78 & lr-AdamW/pg1‚0.0002734353020169356 !Ù(ɇà & lr-AdamW/pg1‚0.0002734353020169356 & lr-AdamW/pg2‚0.0002734353020169356 trainer/global_step‚999 % _timestamp‚1.7843214058994792e+09 _runtime‚1659.274070957 _step‚78N˜ãñ2ˆ- _wandbruntime‚1659  _runtime‚1659ÛÙ5Ö'‰ÐO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78982750233600 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937039904849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286681468006912 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 938644345856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286682839569408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77774751625216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1207964071424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 935695304531 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8639258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285620109873152 trainer/global_step‚999 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2830179 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286681468006912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937298956800 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5394969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8629286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8639258 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285744195223552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8474377 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5395219 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2829934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286682839569408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2565040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚ 1659.27542909 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8474383 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8474377 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285620084445184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17103663 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 # _timestamp‚1.78432140590067e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8474383 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937298956800 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78982347605504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 938644345856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2565035 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648„­d%Ú'ŠÔO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8639258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285620109873152 trainer/global_step‚999 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2830179 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286681468006912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937298956800 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5394969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8629286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8639258 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285744195223552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8474377 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5395219 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2829934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286682839569408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2565040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚ 1659.27542909 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8474383 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8474377 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285620084445184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17103663 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 # _timestamp‚1.78432140590067e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8474383 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937298956800 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78982347605504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 938644345856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2565035 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78982750233600 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937039904849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286681468006912 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 938644345856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286682839569408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77774751625216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1207964071424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 935695304531 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚999 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8482589 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285897966223360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8482589 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 1660.66218228 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2832917 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚80 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79061439655936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285897991651328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 939569388544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1209167093248 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 937963371207 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286960393772544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77851869941760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5400256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 AÀTØY8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8647750 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843214072873492e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286022169780224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286834584994137 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286961765342208 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8637764 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5400500 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 939569388544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2567583 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664>~à>æ&àM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77852238032896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286961765342208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 936618770777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843214072873492e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286022169780224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286834584994137 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286961765342208 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8637764 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5400500 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 939569388544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2567583 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚999 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8482589 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285897966223360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8482589 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 1660.66218228 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2832917 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚80 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79061439655936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285897991651328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 939569388544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1209167093248 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 937963371207 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286960393772544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77851869941760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5400256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8647750 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2692185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9065010 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 984960276480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983614887424 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299514198781952 trainer/global_step‚1049 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2967369 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268111321088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2967612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82916780702208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 983277017957 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300627820443136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 SGDeviceStatsMonitor.on_train_batch_start/allocation.smalÚ$´l_pool.allocated‚9065010 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268076784128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299644205555712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299644231729152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299514173353984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81649072009216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300627820443136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300497475799909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ï)±ÞÝ' ×O VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268076784128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299644205555712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299644231729152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299514173353984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81649072009216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300627820443136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300497475799909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5659802 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2692190 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82917183330304 _runtime‚1737.897018704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8884683 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚83 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2692185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9065010 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 984960276480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983614887424 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299514198781952 trainer/global_step‚1049 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2967369 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268111321088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2967612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82916780702208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 983277017957 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300627820443136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9065010 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128S™³Cƒ2¡- _wandbruntime‚1739  _runtime‚17391‚ä&¢ÞM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81726190325760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269280509440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300776281044443 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299792080560128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚1739.282484942 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9073502 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299922232459264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 trainer/global_step‚1049 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 984200484315 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5664828 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2694728 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9073502 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5665070 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82995873456128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984539922944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2970337 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8892889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2694733 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 985885319168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269315039232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299922206285824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300906746208768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2970100 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82995470835200P Fè&£âM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300906746208768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2970100 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82995470835200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81726190325760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269280509440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300776281044443 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299792080560128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚1739.282484942 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9073502 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299922232459264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 trainer/global_step‚1049 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 984200484315 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5664828 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2694728 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9073502 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5665070 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82995873456128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984539922944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2970337 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8892889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2694733 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 985885319168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269315039232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299922206285824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0T*è\2¤- _wandbruntime‚1739  _runtime‚1739'˜]–2¥ù 2loss/SRqqt1/assignment_loss‚0.7366004811624454 /loss/FRt2/detection_loss‚0.6478071378216276 1loss/SRqqt2/detection_loss‚1.3803570357092472 .loss/FBt2/detection_loss‚3.293707326054573 _runtime‚1739.368421918 0loss/FRt1/assignment_loss‚2.0145907003990344 0loss/SRqqt2/assignment_loss‚1.13748934032347 1loss/FBt2/assignment_loss‚0.23954066832946544 epoch‚2 %loss/total_loss‚16.10907872304745 0loss/FRt2/assignment_loss‚3.3966102349166136 1loss/FBt1/assignment_loss‚0.11589697694556418 .loss/FRt1/detection”æ£\Í_loss‚1.390035478170182 1loss/SRqqt1/detection_loss‚0.7119126417296074 trainer/global_step‚1049 # _timestamp‚1.78432148599405e+09 .loss/FBt1/detection_loss‚1.044530701485619 _step‚85ªH:¦ý .loss/FBt2/detection_loss‚3.293707326054573 _runtime‚1739.368421918 0loss/FRt1/assignment_loss‚2.0145907003990344 0loss/SRqqt2/assignment_loss‚1.13748934032347 1loss/FBt2/assignment_loss‚0.23954066832946544 epoch‚2 %loss/total_loss‚16.10907872304745 0loss/FRt2/assignment_loss‚3.3966102349166136 1loss/FBt1/assignment_loss‚0.11589697694556418 .loss/FRt1/detection_loss‚1.390035478170182 1loss/SRqqt1/detection_loss‚0.7119126417296074 trainer/global_step‚1049 # _timestamp‚1.78432148599405e+09 .loss/FBt1/detection_loss‚1.044530701485619 _step‚85 2loss/SRqqt1/assignment_loss‚0.7366004811624454 /loss/FRt2/detection_loss‚0.6478071378216276 1loss/SRqqt2/detection_loss‚1.3803570357092472Uztû)•§‚(:Š ΫêÒÓ’“¬ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚246.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.3552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f1(. ¨‚(: ΫêÒ؈ú¯ disk.sda2.out‚0& proc.memory.availableMB‚ 973.515625) proc.memory.percent‚17.49134063720703 network.recv‚469377 proc.memory.rssMB‚ 2865.78125$ disk./.usageGB‚577.4839286804199 disk.nvme0n1.in‚ 3293.6015625 disk.sda.in‚0 disk.rbd2.out‚ 7.05078125# disk.dm-0.out‚29025.52294921875 cpu‚25.46127276184204 proc.cpu.threads‚17) disk./.usagePercent‚38.75543099503288 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚29025.52294921875 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.sda.out‚ 107.94140625# disk.rbd1.out‚13153.07080078125 disk.rbd0.out‚0$ memory_percent‚94.05813217163086 disk.dm-1.in‚0 disk.dm-1.out‚ 107.94140625" disk.nvme0c0n1.in‚ 3293.6015625 disk.dm-0.in‚ 3293.6015625 disk.sda3.in‚0 disk.sda3.out‚ 107.94140625& disk.nvme0n1.out‚29025.52294921875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0.046875 network.sent‚ 1.794642e+06 disk.drbd2165.in‚0sK‚ÕŸ©‚(:” Ý«êҽ̡® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚262.144' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.428799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0׫—ª‚(:õ Ý«êÒ¯üÖ® proc.memory.rssMB‚ 2865.78125 network.sent‚ 1.857118e+06 disk.sda.in‚0 disk.sda2.in‚0! disk.dm-0.out‚29031.732421875 cpu‚25.46065370925592 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.in‚0$ memory_percent‚94.05412673950195 network.recv‚474066& disk.nvme0c0n1.out‚29031.732421875 disk.nvme0n1.in‚ 3422.71875 disk.sda2.out‚0 disk.sda1.out‚0$ disk./.usageGB‚577.4839897155762 disk.nvme0c0n1.in‚ 3422.71875$ disk.nvme0n1.out‚29031.732421875 disk.sda.out‚ 108.58984375 disk.rbd1.in‚0.046875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 7.05078125 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 108.58984375 disk.sda3.out‚ 108.58984375 disk.sda3.in‚0# disk.rbd1.out‚13159.82080078125 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3422.71875& proc.memory.availableMB‚ 974.171875) proc.memory.percent‚17.49134063720703) disk./.usagePercent‚38.75543509115343r´‹”«‚(:‰ ì«êҶǺ­ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚284.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pàï=¬‚(:† ì«êÒðúί disk.rbd2.out‚ 7.50927734375# disk.dm-0.out‚29039.38427734375 disk.sda1.out‚0 disk.rbd1.in‚0.046875 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚577.4846458435059 disk.dm-0.in‚ 3423.55078125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 109.1015625# disk.rbd1.out‚13290.41455078125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.in‚0) proc.memory.percent‚17.49134063720703 cpu‚25.46011923757001( disk.nvme0c0n1.out‚29039.38427734375* disk./.usagePercent‚38.755479124449316 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3423.55078125 disk.sda2.in‚0# memory_percent‚94.0556287765503! disk.nvme0n1.in‚ 3423.55078125& disk.nvme0n1.out‚29039.38427734375 disk.rbd0.in‚0 network.sent‚ 1.862176e+06 disk.sda3.out‚ 109.1015625 disk.sda2.out‚0 network.recv‚477378 disk.dm-1.out‚ 109.1015625 disk.rbd2.in‚0.015625( proc.memory.availableMB‚ 973.92578125 proc.memory.rssMB‚ 2865.78125, Á ž­‚(:“ û«êÒþóǬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚278.633' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.72659999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦}’6 ®‚(:€ û«êÒ華® disk.sda2.out‚0# disk.rbd1.out‚13298.53955078125 network.recv‚480684 disk.dm-1.out‚ 109.890625 proc.memory.rssMB‚ 2865.78125 cpu‚25.45951833848617* disk./.usagePercent‚38.755413330513015 disk.rbd2.in‚0.015625( disk.nvme0c0n1.out‚29051.78271484375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚29051.78271484375 disk.rbd1.in‚0.046875 network.sent‚ 1.86719e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 109.890625$ memory_percent‚94.05534267425537) proc.memory.percent‚17.49134063720703# disk.dm-0.out‚29051.78271484375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚577.4836654663086 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 3424.5078125 disk.nvme0n1.in‚ 3424.5078125 proc.cpu.threads‚17( proc.memory.availableMB‚ 973.97265625 disk.rbd2.out‚ 7.50927734375 disk.dm-0.in‚ 3424.5078125 disk.sda.in‚0 disk.sda.out‚ 109.890625DÉÑ»Ÿ¯‚(:” Š¬êÒüËÞ« gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.398599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë0Ë°‚(:‹ Š¬êÒ¥‚Ý® disk.drbd2165.in‚0 disk.sda3.out‚ 110.25390625' proc.memory.availableMB‚ 974.2265625 disk.sda.out‚ 110.25390625 disk.rbd2.out‚ 8.35595703125 disk.dm-0.in‚ 3424.63671875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3424.63671875* disk./.usagePercent‚38.755423314806855& disk.nvme0n1.out‚29055.00732421875! proc.memory.rssMB‚ 2865.296875 disk.rbd0.out‚0# disk.dm-0.out‚29055.00732421875 disk.sda3.in‚0* proc.memory.percent‚17.488384246826172 network.recv‚483209 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.in‚0 network.sent‚ 1.871298e+06 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚25.458745499450295 proc.cpu.threads‚17# disk./.usageGB‚577.483814239502 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 110.25390625( disk.nvme0c0n1.out‚29055.00732421875! disk.nvme0n1.in‚ 3424.63671875 disk.rbd1.in‚0.046875# disk.rbd1.out‚13413.14892578125$ memory_percent‚94.05379295349121D+r”±‚(:‰ ™¬êҡ뮬 gpu.0.gpu‚55.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚112.273' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4546 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìµ}²‚(:„ ™¬êÒò¢ñ­ disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 110.5234375 disk.sda2.in‚0$ memory_percent‚94.05550956726074 cpu‚25.458155067294253# disk.nvme0c0n1.in‚ 3526.53515625 network.sent‚ 1.876969e+06 network.recv‚486625 disk.dm-0.in‚ 3526.53515625 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 110.5234375 proc.memory.rssMB‚ 2865.78125 disk.drbd2165.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.49134063720703 proc.cpu.threads‚17" disk.dm-0.out‚29065.1494140625' disk.nvme0c0n1.out‚29065.1494140625% disk.nvme0n1.out‚29065.1494140625 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda2.out‚0# disk.rbd1.out‚13421.27392578125 disk.sda1.out‚0 disk.dm-1.out‚ 110.5234375$ disk./.usageGB‚577.4837341308594 disk.rbd2.out‚ 8.35595703125 disk.drbd2165.out‚0' proc.memory.availableMB‚ 973.9453125! disk.nvme0n1.in‚ 3526.53515625* disk./.usagePercent‚38.755417938648634±f)32³- _wandbruntime‚1816  _runtime‚1816¡%fêÆ´À & lr-AdamW/pg1‚0.0002732072310965933 & lr-AdamW/pg2‚0.0002732072310965933 trainer/global_step‚1099 % _timestamp‚1.7843215631450486e+09 _runtime‚1816.519705095 _step‚86#p€#浀 & lr-AdamW/pg1‚0.0002732072310965933 & lr-AdamW/pg2‚0.0002732072310965933 trainer/global_step‚1099 % _timestamp‚1.7843215631450486e+09 _runtime‚1816.519705095 _step‚86V-ÿ‚ì2¶- _wandbruntime‚1816  _runtime‚1816/Ç æà'·ÚO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3104785 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86851226089984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9480790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313408287690752 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9480790 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313544268234752 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9294983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313544242061312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314575544441856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5924381 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1328236324864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313408262262784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1029930818048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1816.520904307 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314574172879360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9490762 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1031276207104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚87 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9490762 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314437801821817 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9294977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313544268234752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314574172879360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1031276207104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85523392393216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1028169530747 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1029930818048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9294983 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85523024302080 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843215631461754e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3105041 ð›Æä'¸ÞO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313408262262784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1029930818048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1816.520904307 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314574172879360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9490762 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1031276207104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚87 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9490762 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314437801821817 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9294977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313544268234752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314574172879360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1031276207104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85523392393216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1028169530747 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1029930818048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9294983 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85523024302080 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843215631461754e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3105041 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3104785 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86851226089984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9480790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313408287690752 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9480790 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313544268234752 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9294983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313544242061312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314575544441856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5924381 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1328236324864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0WXoE2¹- _wandbruntime‚1817  _runtime‚1817ûyµÆºãM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9489268 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313822242791424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.sÕ©˜#mall_pool.freed‚ 1329392062976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314715237037953 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313686144040960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9303183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1329428689920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314853098644992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3107775 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9489268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9303189 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313686169468928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _step‚88 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5929402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313822242791424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85600878800896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1030855853568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1030855853568 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9303183 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5929658 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1029092996993 % _timestamp‚1.7843215645313895e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚1099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85600510709760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314854470214656 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9303189 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1030437597423 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1032201249792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1032201249792<ùHí&»çM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314853098644992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3107775 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9489268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9303189 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313686169468928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _step‚88 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5929402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313822242791424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85600878800896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1030855853568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1030855853568 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9303183 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5929658 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1029092996993 % _timestamp‚1.7843215645313895e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚1099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85600510709760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314854470214656 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9303189 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1030437597423 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1032201249792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1032201249792 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9489268 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313822242791424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1329392062976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314715237037953 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313686144040960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9303183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1329428689920X„ÁÄš2¼- _wandbruntime‚1817  _runtime‚1817‚Už½û /loss/FBt1/detection_loss‚0.8736070545819544 &loss/total_loss‚14.101615348774153 epoch‚2 2loss/SRqqt2/assignment_loss‚1.1262309489030282 1loss/FBt1/assignment_loss‚0.07260171795722895 1loss/FBt2/assignment_loss‚0.11186392806932417 .loss/FRt1/detection_loss‚1.478275707690474 /loss/FRt2/detection_loss‚0.6767691165235602 1loss/SRqqt1/detection_loss‚0.7584492563975732 0loss/FRt1/assignment_loss‚1.7482114520959873 /loss/FRt2/assignment_loss‚3.257989778257577 .loss/SRqqt2/detection_loss‚1.3136027230848 trainer/global_step‚1099 $ _timestamp‚1.784321564617521e+09 _runtime‚1817.991850853 _step‚89 2loss/SRqqt1/assignment_loss‚0.7020348151459864 /loss/FBt2/detection_loss‚1.9819788500666617ɧòξÿ /loss/FRt2/detection_loss‚0.6767691165235602 1loss/SRqqt1/detection_loss‚0.7584492563975732 0loss/FRt1/assignment_loss‚1.7482114520959873 /loss/FRt2/assignment_loss‚3.257989778257577 .loss/SRqqt2/detection_loss‚1.3136027230848 trainer/global_step‚1099 $ _timestamp‚1.784321564617521e+09 _runtime‚1817.991850853 _step‚89 2loss/SRqqt1/assignment_loss‚0.7020348151459864 /loss/FBt2/detection_loss‚1.9819788500666617 /loss/FBt1/detection_loss‚0.8736070545819544 &loss/total_loss‚14.101615348774153 epoch‚2 2loss/SRqqt2/assignment_loss‚1.1262309489030282 1loss/FBt1/assignment_loss‚0.07260171795722895 1loss/FBt2/assignment_loss‚0.11186392806932417 .loss/FRt1/detection_loss‚1.478275707690474Y–¾ |ž¿‚(:“ ¨¬êÒ‚훬 gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚291.533' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.30660000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±÷<éÀ‚(:ü ¨¬êÒ»Š¯® disk.rbd0.out‚0 disk.sda1.in‚0 network.recv‚489826& disk.nvme0n1.out‚29073.41064453125( disk.nvme0c0n1.out‚29073.41064453125 disk.rbd1.in‚0.046875 disk.rbd2.out‚ 8.46630859375 disk.dm-1.in‚0 network.sent‚ 1.881611e+06 disk.nvme0n1.in‚ 3527.4921875 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0$ disk./.usageGB‚577.4838714599609 disk.sda3.in‚0# disk.dm-0.out‚29073.41064453125 disk.drbd2165.out‚0# memory_percent‚94.0556287765503 cpu‚25.45753607664678) disk./.usagePercent‚38.75542715491987 disk.dm-1.out‚ 110.71875) proc.memory.percent‚17.49134063720703 disk.sda.out‚ 110.71875 disk.dm-0.in‚ 3527.4921875 disk.sda1.out‚0( proc.memory.availableMB‚ 973.92578125 proc.memory.rssMB‚ 2865.78125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.out‚ 110.71875" disk.nvme0c0n1.in‚ 3527.4921875 disk.sda2.out‚0# disk.rbd1.out‚13550.74267578125øì•a”Á‚(:‰ ·¬êÒ–¸­ gpu.0.gpu‚94.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚343.312' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6624 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê"úý‚(:ò ·¬êÒ–ꊯ# disk.nvme0n1.out‚29079.30078125 cpu‚25.457187740047196 disk.dm-0.in‚ 3527.4921875 disk.dm-0.out‚29079.30078125 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 974.37890625 disk.rbd0.out‚0 disk.dm-1.out‚110.875 disk.sda.out‚110.875* proc.memory.percent‚17.487192153930664 proc.cpu.threads‚17 disk.rbd2.out‚ 8.46630859375 disk.sda1.in‚0 disk.nvme0n1.in‚ 3527.4921875$ disk./.usageGB‚577.4838981628418 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 3527.4921875% disk.nvme0c0n1.out‚29079.30078125$ memory_percent‚94.05286312103271" proc.memory.rssMB‚ 2865.1015625# disk.rbd1.out‚13564.36767578125 network.recv‚494594 disk.rbd2.in‚0.015625 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.046875) disk./.usagePercent‚38.75542894697261 disk.dm-1.in‚0 disk.sda3.out‚110.875 network.sent‚ 1.944837e+06 disk.rbd0.in‚0 disk.sda1.out‚0¹Õ ”Â(:‰ Ƭê҇ѫ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PBÄ‚(:… ƬêÒ‚ô¿® disk.sda2.out‚0# disk.dm-0.out‚29086.82080078125 cpu‚25.456351275573642 disk.sda1.out‚0! proc.memory.rssMB‚ 2865.296875 proc.cpu.threads‚17# disk.rbd1.out‚13651.77392578125 disk.rbd2.out‚ 8.490234375 disk.sda1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.488384246826172$ disk./.usageGB‚577.4838562011719! disk.nvme0n1.in‚ 3528.32421875 disk.rbd1.in‚0.046875 disk.dm-1.out‚ 111.3046875 disk.sda3.in‚0& disk.nvme0n1.out‚29086.82080078125 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 3528.32421875 disk.drbd2165.in‚0& proc.memory.availableMB‚ 973.703125# disk.nvme0c0n1.in‚ 3528.32421875 disk.rbd0.out‚0 disk.dm-1.in‚0$ memory_percent‚94.05698776245117 network.sent‚ 1.949857e+06) disk./.usagePercent‚38.75542613088973 disk.rbd2.in‚0.015625 disk.sda3.out‚ 111.3046875( disk.nvme0c0n1.out‚29086.82080078125 disk.sda.out‚ 111.3046875 disk.rbd0.in‚0 network.recv‚497683iÙ¼·”Å‚(:‰ Õ¬êÒÀÛ•® gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚343.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 åøYúÆ‚(:ï Õ¬êÒȹή disk.rbd2.out‚ 8.490234375 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3584.484375 disk.drbd2165.out‚0 disk.dm-1.out‚ 111.5546875$ memory_percent‚94.05858516693115 proc.memory.rssMB‚ 2865.78125 network.sent‚ 1.954904e+06$ disk./.usageGB‚577.4838371276855 disk.sda3.in‚0# disk.nvme0c0n1.out‚ 29096.890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 29096.890625) proc.memory.percent‚17.49134063720703) disk./.usagePercent‚38.75542485085206! disk.nvme0c0n1.in‚ 3584.484375 disk.nvme0n1.in‚ 3584.484375 disk.sda1.in‚0! disk.nvme0n1.out‚ 29096.890625 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 cpu‚25.45590333479897 network.recv‚501171 disk.sda3.out‚ 111.5546875 disk.sda.out‚ 111.5546875# disk.rbd1.out‚13671.49267578125( proc.memory.availableMB‚ 973.44140625 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0zœ¨/“Ç‚(:ˆ ä¬êÒî•Ô­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚285.045' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.009 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡"s2 È‚(:€ ä¬êÒÙ—š®$ disk./.usageGB‚577.4838523864746 disk.rbd1.in‚0.046875 disk.dm-0.in‚ 3584.484375 disk.sda1.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚29100.7060546875% disk.nvme0n1.out‚29100.7060546875 disk.sda1.in‚0 disk.sda3.out‚ 111.81640625! disk.nvme0c0n1.in‚ 3584.484375 cpu‚25.45521551946484 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 111.81640625 network.recv‚504245# disk.rbd1.out‚13729.78955078125 disk.rbd0.in‚0 disk.rbd2.out‚ 8.55322265625( proc.memory.availableMB‚ 973.41015625 proc.memory.rssMB‚ 2865.78125) disk./.usagePercent‚38.75542587488219 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚29100.7060546875 disk.drbd2165.in‚0) proc.memory.percent‚17.49134063720703 network.sent‚ 1.959927e+06 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 111.81640625 disk.nvme0n1.in‚ 3584.484375 disk.sda.in‚0 proc.cpu.threads‚17$ memory_percent‚94.058775901794437ÕYñ2É- _wandbruntime‚1895  _runtime‚1895*æ¤æÄʾ % lr-AdamW/pg1‚0.000273080072528851 % lr-AdamW/pg2‚0.000273080072528851 trainer/global_step‚1149 % _timestamp‚1.7843216417109177e+09 _runtime‚1895.085576791 _step‚90[ž¥oÈË trainer/global_step‚1149 % _timestamp‚1.7843216417109177e+09 _runtime‚1895.085576791 _step‚90 % lr-AdamW/pg1‚0.000273080072528851 % lr-AdamW/pg2‚0.000273080072528851Z«Lò.2Ì- _wandbruntime‚1895  _runtime‚1895ÙªŒ6ÍÚO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90785959884800 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9916514 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388212570624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328521896878080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328520525315584 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6188766 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2946490 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1076246748672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521896878080 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89397712777216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19621797 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327302351171584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1074406643855 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9705283 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1077592137728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843216417121398e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3242037 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1149 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9906542 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327444304740352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1077592137728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1075751244173 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90785557256704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328520525315584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2946485 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3242276 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚91 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192Y—Ãä'ÎÞO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9916514 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388247107584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328378127843725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328376757815439 _runtime‚1895.086896202 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9906542 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327444278566912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9705277 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9705277 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19611819 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6188522 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89397344686080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9906542 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327444304740352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1077592137728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1075751244173 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90785557256704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328520525315584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2946485 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3242276 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚91 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90785959884800 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9916514 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388212570624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328521896878080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328520525315584 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6188766 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2946490 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1076246748672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521896878080 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89397712777216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19621797 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327302351171584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1074406643855 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9705283 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1077592137728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843216417121398e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3242037 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1149 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6193806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328655563059861 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327722279297024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1389408317952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843216430958652e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1078517180416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1078517180416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3244778 trainer/global_step‚1149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1075330110101 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9925006 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9915020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6194044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89475199184896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2949028 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9713483 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9915020 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1076674710531 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚1896.470674277 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90864642032640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3245011 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90864239411712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328800822650880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1077171784192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327722305470464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89474831093760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327722305470464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89474831093760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6193806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328655563059861 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327722279297024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1389408317952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843216430958652e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1078517180416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1078517180416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3244778 trainer/global_step‚1149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1075330110101 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9925006 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9915020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6194044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89475199184896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2949028 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9713483 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9915020 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1076674710531 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚1896.470674277 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90864642032640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3245011 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90864239411712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328800822650880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1077171784192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10115583 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341344315072512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1122562679296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341196465508352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1120643756963 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342468249314304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342466877751808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342318453865633 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1122562679296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342317083837347 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94720454391808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341196440080384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10115577 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10115577 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6453358 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10332294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1123908068352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341344315072512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10115583 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448421230592 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚1973.702581602 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342466877751808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10332294 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341344341245952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94720053860864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341344341245952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056Ô–Gì'äæO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94720454391808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341196440080384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10115577 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10115577 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6453358 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10332294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1123908068352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341344315072512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10115583 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448421230592 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚1973.702581602 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342466877751808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10332294 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341344341245952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94720053860864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341344341245952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843217203278027e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3073635 trainer/global_step‚1199 _step‚95 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3073640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3379718 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10115583 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341344315072512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1122562679296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341196465508352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1120643756963 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342468249314304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342466877751808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342318453865633 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1122562679296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342317083837347_ÔŸå±2å- _wandbruntime‚1975  _runtime‚1975öµmìGæìM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1123487714816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1121567223209 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1123487714816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10350758 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342745803517440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10123789 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342597259110167 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94799155233792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341622315802624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341622341976064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1122911823639 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10340772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10123783 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6458375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10123783 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94798754710016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1121567223209 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1123487714816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10350758 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342745803517440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10123789 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342597259110167 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94799155233792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341622315802624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341622341976064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1122911823639 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10340772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10123783 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6458375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10123783 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94798754710016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5150533318519592 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8146679401397705 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7286185026168823 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9944843053817749 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5632086396217346 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009123905561864376 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9044886827468872 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3852590024471283 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5693174600601196 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07929381728172302 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7097489833831787 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05114566534757614 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9953571557998657 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5536565780639648 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906987547874451 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6207901239395142 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7097489833831787 6validation_average_jet_accuracy‚0.2874412536621094 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.986603856086731 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930440783500671 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6729962229728699 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4711754322052002 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03487606346607208 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5185955762863159 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02627190761268139 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09910046309232712 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5351935029029846 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09833333641290665 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019922083243727684 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5604463815689087 6"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6912881731987 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 particle/accuracy_3_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9838544130325317 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4335789978504181 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9949694871902466 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12149535864591599 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.664739191532135 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917629957199097 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03224007040262222 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08540026843547821 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9732077717781067 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878970384597778 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010387904942035675 ,jet/accuracy_4_of_4‚0.009931244887411594 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8583426475524902 +jet/accuracy_2_of_2‚0.10897842049598694 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09256695210933685 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10872463136911392 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4729481637477875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982336699962616 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4820670187473297 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6663076281547546 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5882506370544434 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.05268091335892677 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6072530150413513 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.44149136543273926 epoch‚2 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5841252207756042 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5342501401901245 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.25564756989479065 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.668334543704986ÛKÉå˜6 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9900356531143188 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.03337731957435608 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5520833730697632 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33796295523643494 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 *jet/accuracy_2_of_3‚0.4186772406101227 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9938237071037292 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5580376386642456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5235276818275452 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.03337731957435608 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6138639450073242 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897609949111938 trainer/global_step‚1199 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9948132634162903 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0334685854613781 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.566425621509552 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0036397906951606274 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 *jet/accuracy_2_of_4‚0.6902403831481934 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32666298747062683 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06202644854784012 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06654929369688034 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41435185074806213 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8361932039260864 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚98 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20139794051647186 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.566425621509552 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6975296139717102 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_4_of_4‚0.9170811772346497 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.045665204524993896 particle/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.40120169520378113 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11121522635221481 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03728271275758743 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07825861126184464 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.611986517906189 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9156266450881958 *jet/accuracy_1_of_4‚0.9329006671905518 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06460706144571304 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6936718225479126 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992766261100769 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6973358988761902 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006764375604689121 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9933122992515564 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3626177906990051 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05823775753378868 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9921366572380066 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15917815268039703 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5620312690734863 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 _runtime‚2001.206911078 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8720892667770386 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.6059189438819885 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.46472516655921936 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09910046309232712 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.265625 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48084163665771484 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3251679241657257 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5888177156448364 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_3‚0.8802034258842468 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987450361251831 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_3_of_4‚0.06800855696201324 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45374998450279236 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1010148748755455 $ _timestamp‚1.784321747831646e+09 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9944843053817749 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24893075227737427 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6188459396362305 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031Àì0ü=ñö{ <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09256695210933685 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10872463136911392 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4729481637477875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982336699962616 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4820670187473297 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6663076281547546 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5882506370544434 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.05268091335892677 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6072530150413513 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.44149136543273926 epoch‚2 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5841252207756042 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5342501401901245 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.25564756989479065 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6683345437049866 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9900356531143188 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.03337731957435608 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5520833730697632 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33796295523643494 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 *jet/accuracy_2_of_3‚0.4186772406101227 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9938237071037292 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5580376386642456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5235276818275452 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.03337731957435608 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6138639450073242 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897609949111938 trainer/global_step‚1199 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9948132634162903 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0334685854613781 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.566425621509552 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0036397906951606274 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 *jet/accuracy_2_of_4‚0.6902403831481934 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32666298747062683 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06202644854784012 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06654929369688034 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41435185074806213 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8361932039260864 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚98 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20139794051647186 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.566425621509552 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6975296139717102 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_4_of_4‚0.9170811772346497 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.045665204524993896 particle/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.40120169520378113 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11121522635221481 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03728271275758743 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07825861126184464 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.611986517906189 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9156266450881958 *jet/accuracy_1_of_4‚0.9329006671905518 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06460706144571304 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6936718225479126 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992766261100769 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6973358988761902 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006764375604689121 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9933122992515564 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3626177906990051 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05823775753378868 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9921366572380066 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15917815268039703 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5620312690734863 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 _runtime‚2001.206911078 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8720892667770386 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.6059189438819885 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.46472516655921936 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09910046309232712 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.265625 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48084163665771484 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3251679241657257 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5888177156448364 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_3‚0.8802034258842468 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987450361251831 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_3_of_4‚0.06800855696201324 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45374998450279236 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1010148748755455 $ _timestamp‚1.784321747831646e+09 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9944843053817749 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24893075227737427 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6188459396362305 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11321679502725601 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08605298399925232 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9904585480690002 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.615362823009491 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8390709757804871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22133854031562805 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1067049503326416 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09359046071767807 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5923054814338684 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3515625 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6577097773551941 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5007638931274414 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6245147585868835 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.11641354858875275 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.046130165457725525 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36612194776535034 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6823147535324097 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 +jet/accuracy_1_of_1‚0.23262716829776764 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887576103210449 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_3‚0.9994933605194092 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916895031929016 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.047145742923021317 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018350420519709587 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.5864547491073608 *loss/val_total_loss‚13.556967611138784 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.656897246837616 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6877185106277466 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04927653819322586 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08538056910037994 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6918298602104187 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5150533318519592 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8146679401397705 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7286185026168823 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9944843053817749 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5632086396217346 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009123905561864376 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9044886827468872 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3852590024471283 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5693174600601196 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07929381728172302 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7097489833831787 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05114566534757614 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9953571557998657 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5536565780639648 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906987547874451 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6207901239395142 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7097489833831787 6validation_average_jet_accuracy‚0.2874412536621094 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.986603856086731 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930440783500671 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6729962229728699 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4711754322052002 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03487606346607208 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5185955762863159 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02627190761268139 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09910046309232712 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5351935029029846 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09833333641290665 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019922083243727684 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5604463815689087 6"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6912881731987 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 particle/accuracy_3_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9838544130325317 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4335789978504181 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9949694871902466 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12149535864591599 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.664739191532135 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917629957199097 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03224007040262222 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08540026843547821 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9732077717781067 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878970384597778 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010387904942035675 ,jet/accuracy_4_of_4‚0.009931244887411594 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8583426475524902 +jet/accuracy_2_of_2‚0.10897842049598694b“dµ4•ò‚(:Š Ü­êÒÊÔ‡­ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚345.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.1468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ’ó¤ó‚(:ƒ Ü­êÒíÒ‰¯$ memory_percent‚88.12229633331299 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 120.265625# disk.nvme0c0n1.in‚ 3693.09765625 disk.sda.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 120.265625) proc.memory.percent‚17.60127544403076 network.sent‚ 2.117011e+06 disk.sda2.in‚0# disk.rbd1.out‚14315.10205078125 disk.rbd2.out‚ 9.2587890625 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 3693.09765625) disk./.usagePercent‚38.82163198331207 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 3693.09765625) proc.memory.availableMB‚ 1946.04296875 disk.rbd1.in‚0.046875 cpu‚25.48080892382376 proc.cpu.threads‚17# proc.memory.rssMB‚ 2883.79296875$ disk./.usageGB‚578.4703712463379 disk.rbd0.in‚0% disk.nvme0n1.out‚30191.8935546875 disk.sda.out‚ 120.265625 network.recv‚532750" disk.dm-0.out‚30191.8935546875 disk.sda1.out‚0' disk.nvme0c0n1.out‚30191.8935546875‘;Ÿô‚(:” ë­êÒЮΫ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚267.508' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.501599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÜà õ‚(: ë­êÒŒôË­ network.recv‚537297 disk.dm-1.in‚0 disk.rbd1.in‚0.046875# disk.rbd1.out‚14483.32080078125$ disk./.usageGB‚578.4667434692383 disk.dm-1.out‚ 121.26171875 disk.nvme0n1.in‚ 3694.0546875 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.out‚ 121.26171875 disk.rbd2.out‚ 9.2587890625& disk.nvme0c0n1.out‚30206.865234375) disk./.usagePercent‚38.82138852014702 disk.rbd0.in‚0 disk.dm-0.in‚ 3694.0546875 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1945.5703125 disk.rbd0.out‚0) proc.memory.percent‚17.60416030883789 cpu‚25.480012074076697 proc.cpu.threads‚17 network.sent‚ 2.174125e+06 disk.rbd2.in‚0.015625! disk.dm-0.out‚30206.865234375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 121.26171875" disk.nvme0c0n1.in‚ 3694.0546875$ disk.nvme0n1.out‚30206.865234375 disk.sda2.in‚0$ memory_percent‚88.12518119812012! proc.memory.rssMB‚ 2884.265625žWf“ö‚(:ˆ ú­êÒŽŸÇ® gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òø”÷‚(:ˆ ú­êÒáÉ™¯$ disk./.usageGB‚578.4673156738281% disk.nvme0n1.out‚31192.9248046875 disk.sda2.out‚0" disk.dm-0.out‚31192.9248046875 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚31192.9248046875 disk.sda.out‚ 123.43359375 disk.rbd2.in‚0.015625 disk.sda3.out‚ 123.43359375 disk.dm-1.in‚0) disk./.usagePercent‚38.82142692127715 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 3694.88671875) proc.memory.availableMB‚ 1945.48828125 network.sent‚ 2.179731e+06 network.recv‚540209 cpu‚25.47938946567108 disk.sda.in‚0 disk.dm-1.out‚ 123.43359375 disk.sda3.in‚0! disk.nvme0n1.in‚ 3694.88671875! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789# disk.nvme0c0n1.in‚ 3694.88671875# disk.rbd1.out‚14543.07080078125 disk.rbd2.out‚ 9.79541015625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚88.12568187713623 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.drbd2165.in‚0 proc.cpu.threads‚17v|Š”ø‚(:‰ ‰®êÒ„ê•­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚271.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï<È ù‚(:€ ‰®êÒ·­ô®! disk.nvme0c0n1.in‚ 3755.984375 disk.rbd1.in‚0.046875# memory_percent‚88.1272554397583* disk./.usagePercent‚38.821446633857285 proc.cpu.threads‚17 network.sent‚ 2.184216e+06 disk.sda3.in‚0 disk.sda3.out‚ 124.41015625 disk.sda2.out‚0 disk.dm-1.out‚ 124.41015625! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789& disk.nvme0c0n1.out‚31200.529296875 disk.sda2.in‚0# disk.rbd1.out‚14682.53955078125 disk.dm-0.in‚ 3755.984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 network.recv‚543351$ disk./.usageGB‚578.4676094055176 disk.nvme0n1.in‚ 3755.984375$ disk.nvme0n1.out‚31200.529296875 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625! disk.dm-0.out‚31200.529296875 disk.sda.out‚ 124.41015625) proc.memory.availableMB‚ 1945.23046875 disk.rbd0.out‚0 disk.rbd2.out‚ 9.79541015625 disk.drbd2165.in‚0 cpu‚25.478584111336104®62lžú‚(:“ ˜®êÒ¤ù¢¬ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.268' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.25359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i&û‚(:Š ˜®êÒ“œþ® disk.sda.out‚ 126.13671875 network.sent‚ 2.189338e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.046875 disk.dm-1.out‚ 126.13671875# disk.rbd1.out‚14690.66455078125 network.recv‚546711) disk./.usagePercent‚38.82122237125729 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3756.94140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ disk./.usageGB‚578.4642677307129# disk.dm-0.out‚31213.27880859375! disk.nvme0n1.in‚ 3756.94140625& disk.nvme0n1.out‚31213.27880859375 disk.sda.in‚0 cpu‚25.477899993361902 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 3756.94140625 disk.sda2.out‚0! proc.memory.rssMB‚ 2884.265625 proc.cpu.threads‚17 disk.rbd2.out‚ 9.8388671875 disk.sda3.in‚0( disk.nvme0c0n1.out‚31213.27880859375$ memory_percent‚88.12575340270996( proc.memory.availableMB‚ 1945.4765625 disk.sda1.out‚0 disk.sda3.out‚ 126.13671875) proc.memory.percent‚17.604160308837899EÂd2ü- _wandbruntime‚2083  _runtime‚2083­ÕämÆýÀ _runtime‚2083.741743118 _step‚99 & lr-AdamW/pg1‚0.0002727995500134856 & lr-AdamW/pg2‚0.0002727995500134856 trainer/global_step‚1249 % _timestamp‚1.7843218303670533e+09Xfi,ÊþÄ & lr-AdamW/pg1‚0.0002727995500134856 & lr-AdamW/pg2‚0.0002727995500134856 trainer/global_step‚1249 % _timestamp‚1.7843218303670533e+09 _runtime‚2083.741743118 _step‚99cM2[.2ÿ- _wandbruntime‚2083  _runtime‚2083é௻€ ãO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1172710001508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3551535 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10657889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358916463349760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10827004 trainer/global_step‚1249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448Vßq. 192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360090517951590 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359072359186432 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98710394284032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6760986 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3551283 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97194465792000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6761243 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089147923300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚2083.743467313 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10817031 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _step‚100 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10827004 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360247054236160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10817031 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚359072359186432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1176066612736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174054601830 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3209708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515928492032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359072333012992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚2083.743467313 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10817031 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _step‚100 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10827004 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360247054236160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10817031 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚359072359186432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1176066612736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174054601830 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3209708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515928492032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359072333012992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360090517951590 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359072359186432 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98710394284032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6760986 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3551283 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97194465792000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6761243 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089147923300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517136256000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10825509 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 2085.13060343 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚1249 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6766542 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98788685835264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10666095 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3554050 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360367953167722 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1174978068188 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10835496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1175646258688 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3554291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359194345127936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359350359916544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1517101726720 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10666095 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359350333743104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21491598 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6766296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360367953167722 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1174978068188 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10835496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1175646258688 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3554291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359194345127936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359350359916544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_poÊsÔ ol.freed‚ 1517101726720 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10666095 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359350333743104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21491598 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6766296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517136256000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10825509 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 2085.13060343 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚1249 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6766542 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98788685835264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10666095 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640ež’¢2‡ - _wandbruntime‚2085  _runtime‚2085;‰ˆ  1loss/SRqqt1/detection_loss‚0.7344685936825756 &loss/total_loss‚12.960430208770404 _runtime‚2085.216212197 _step‚102 2loss/SRqqt2/assignment_loss‚1.0997189033818067 /loss/FRt1/detection_loss‚1.3373293027594995 epoch‚3 0loss/FRt1/assignment_loss‚1.7195711595763405 2loss/SRqqt1/assignment_loss‚0.6785440779036874 /loss/FRt2/detection_loss‚0.6474993114266545 1loss/SRqqt2/detection_loss‚1.3297304499852836 /loss/FBt2/detection_loss‚1.3137735836207867 trainer/global_step‚1249 0loss/FBt2/assignment_loss‚0.0573752017080551 /loss/FBt1/detection_loss‚0.7899720424301026 % _timestamp‚1.7843218318418727e+09 0loss/FRt2/assignment_loss‚3.1609422388592283 1loss/FBt1/assignment_loss‚0.09150534343638288ëÚ—  ‰ … 0loss/FRt2/assignment_loss‚3.1609422388592283 1loss/FBt1/assignment_loss‚0.09150534343638288 1loss/SRqqt1/detection_loss‚0.7344685936825756 &loss/total_loss‚12.960430208770404 _runtime‚2085.216212197 _step‚102 2loss/SRqqt2/assignment_loss‚1.0997189033818067 /loss/FRt1/detection_loss‚1.3373293027594995 epoch‚3 0loss/FRt1/assignment_loss‚1.7195711595763405 2loss/SRqqt1/assignment_loss‚0.6785440779036874 /loss/FRt2/detection_loss‚0.6474993114266545 1loss/SRqqt2/detection_loss‚1.3297304499852836 /loss/FBt2/detection_loss‚1.3137735836207867 trainer/global_step‚1249 0loss/FBt2/assignment_loss‚0.0573752017080551 /loss/FBt1/detection_loss‚0.7899720424301026 % _timestamp‚1.7843218318418727e+09f9ø·?”Š ‚(:‰ ¶®êÒè…ܬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.138' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô†`†/‹ ‚(:¤ ¶®êÒè¯Û® disk.rbd2.out‚ 10.5322265625 disk.dm-0.in‚ 3757.7734375 disk.sda2.out‚0 cpu‚25.476689173464063$ disk./.usageGB‚578.4670753479004 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875( disk.nvme0c0n1.out‚31239.35107421875 disk.sda.in‚ 0.01171875 network.sent‚ 2.246714e+06 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 3757.7734375" proc.memory.rssMB‚ 2883.7734375 network.recv‚554351 disk.rbd0.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0.046875) proc.memory.percent‚17.60115623474121) disk./.usagePercent‚38.82141079280249 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 130.83203125$ memory_percent‚94.05934810638428 disk.rbd0.out‚0& disk.nvme0n1.out‚31239.35107421875# disk.rbd1.out‚14926.60205078125( proc.memory.availableMB‚ 973.31640625# disk.dm-0.out‚31239.35107421875 proc.cpu.threads‚17 disk.dm-1.out‚ 130.83203125 disk.nvme0n1.in‚ 3757.7734375 disk.sda.out‚ 130.83203125 disk.sda2.in‚0c„G”Œ ‚(:‰ Å®êÒ¶ÔÌ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô^Ö* ‚(:“ Å®êÒ†Š®$ disk./.usageGB‚578.4697341918945$ disk.nvme0n1.out‚31257.865234375 disk.sda.in‚ 0.01171875& disk.nvme0c0n1.out‚31257.865234375 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 136.484375 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789 network.sent‚ 2.299121e+06 disk.nvme0n1.in‚ 3823.4140625 disk.sda3.out‚ 136.484375 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 3823.4140625 disk.sda.out‚ 136.484375 disk.sda2.in‚0# disk.rbd1.out‚15031.36767578125$ memory_percent‚94.05860900878906 network.recv‚558679) disk./.usagePercent‚38.82158923005386 disk.dm-1.in‚ 0.01171875! disk.dm-0.out‚31257.865234375 disk.drbd2165.out‚0$ proc.memory.availableMB‚973.4375 cpu‚25.476049608156313 disk.rbd0.out‚0 disk.rbd2.out‚ 10.5322265625 disk.dm-0.in‚ 3823.4140625 disk.sda1.in‚0ëʆ”Ž ‚(:‰ Ô®êҼϬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚248.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¶ëÞ# ‚(:˜ Ô®êÒֱǮ disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 3823.4140625# disk.rbd1.out‚15266.57080078125 disk.rbd0.in‚0 disk.dm-0.in‚ 3823.4140625! disk.dm-0.out‚31270.330078125$ proc.memory.availableMB‚973.4375 disk.sda.out‚ 138.58203125$ memory_percent‚94.05860900878906 proc.cpu.threads‚17 network.sent‚ 2.304175e+06 network.recv‚561810 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.out‚ 138.58203125 cpu‚25.475509425225784) disk./.usagePercent‚38.82128381306551 disk.rbd2.out‚ 10.662109375) proc.memory.percent‚17.60416030883789 disk.dm-1.out‚ 138.58203125 disk.nvme0n1.in‚ 3823.4140625 disk.rbd1.in‚0.046875$ disk./.usageGB‚578.4651832580566 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875& disk.nvme0c0n1.out‚31270.330078125$ disk.nvme0n1.out‚31270.330078125 disk.sda2.out‚0 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda2.in‚0! proc.memory.rssMB‚ 2884.265625Ncߟ ‚(:” ã®êÒüÊé­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚261.613' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.322599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÿ{L0‘ ‚(:¥ ã®êÒ©¸¸®$ disk./.usageGB‚578.4625473022461 cpu‚25.474759962018005* disk./.usagePercent‚38.821106911859346 disk.dm-0.in‚ 3824.24609375 disk.rbd1.in‚0.046875$ memory_percent‚94.06008720397949 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚31287.04541015625 disk.sda.out‚ 140.12890625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 140.12890625 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0.015625 network.recv‚565519 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 140.12890625 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 973.1953125# disk.dm-0.out‚31287.04541015625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3824.24609375! disk.nvme0n1.in‚ 3824.24609375& disk.nvme0n1.out‚31287.04541015625) proc.memory.percent‚17.60416030883789 proc.cpu.threads‚17 network.sent‚ 2.309253e+06 disk.sda1.in‚0 disk.sda.in‚ 0.01171875 disk.rbd2.out‚ 10.888671875! proc.memory.rssMB‚ 2884.265625# disk.rbd1.out‚15343.93017578125 disk.rbd0.in‚0ê¥o˜+’ ‚(:  ò®êÒñòÝ® network.recv‚568579$ memory_percent‚94.05856132507324! disk.nvme0c0n1.in‚ 3825.203125 disk.sda2.out‚0) disk./.usagePercent‚38.82111075197236 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚ 10.904296875 cpu‚25.474239689571423 disk.sda2.in‚0 disk.dm-0.in‚ 3825.203125 disk.drbd2165.in‚0 disk.sda3.out‚ 140.84765625 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625# disk.dm-0.out‚31294.97802734375 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 network.sent‚ 2.314296e+06$ disk./.usageGB‚578.4626045227051( disk.nvme0c0n1.out‚31294.97802734375 disk.sda.out‚ 140.84765625 disk.sda1.in‚0 disk.dm-1.out‚ 140.84765625# proc.memory.rssMB‚ 2883.77734375 disk.nvme0n1.in‚ 3825.203125& disk.nvme0n1.out‚31294.97802734375 disk.sda.in‚ 0.01171875# disk.rbd1.out‚15497.46142578125 disk.rbd0.in‚0' proc.memory.availableMB‚ 973.4453125) proc.memory.percent‚17.60118007659912 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0V×(5““ ‚(:ˆ ò®êÒÛäÍ® gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚159.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n%0T2” - _wandbruntime‚2162  _runtime‚2162šÊÉ• à ' lr-AdamW/pg1‚0.00027264620408329945 ' lr-AdamW/pg2‚0.00027264620408329945 trainer/global_step‚1299 % _timestamp‚1.7843219092865033e+09 _runtime‚2162.661153025 _step‚1036ŒB Í– Ç _runtime‚2162.661153025 _step‚103 ' lr-AdamW/pg1‚0.00027264620408329945 ' lr-AdamW/pg2‚0.00027264620408329945 trainer/global_step‚1299 % _timestamp‚1.7843219092865033e+09gR©2— - _wandbruntime‚2162  _runtime‚2162KÚ$sí'˜ çO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11242783 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7025817 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚1299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3336853 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374193406672384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11068189 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11068183 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374029473945208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372972395692032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1576038882304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3336858 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1576073418752 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11068183 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374194778235392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11252756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374193406672384 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11252756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102644859594752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372972369518592 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372810552258560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374194778235392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374030843973498 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1220291714938 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3688959 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372972395692032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1576038882304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3336858 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1576073418752 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11068183 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374194778235392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11252756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374193406672384 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11252756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102644859594752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372972369518592 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372810552258560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374194778235392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374030843973498 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1220291714938 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3688959 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097hxž¥ 2š - _wandbruntime‚2164  _runtime‚2164ãd:ó÷&› ñM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1221215181296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11251261 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1223307586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374308279189630 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373250396422144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3339396 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374473704008192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7031091 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374473704008192 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373250370248704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101146272583680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374309649218032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11076389 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102723138622464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1577268659200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102723541242880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1577234129920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7030842 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373250396422144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚1299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1221962189312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374472332438016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11076389 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11076395 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11076395 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373088434036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843219106772277e+09-`òû&œ õM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374472332438016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11076389 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11076395 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11076395 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373088434036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843219106772277e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1221215181296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11251261 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1223307586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374308279189630 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373250396422144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3339396 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374473704008192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7031091 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374473704008192 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373250370248704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101146272583680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374309649218032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11076389 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102723138622464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1577268659200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102723541242880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1577234129920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7030842 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373250396422144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚1299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1221962189312iƒ_Ï2 - _wandbruntime‚2164  _runtime‚2164øUž þ 1loss/SRqqt1/detection_loss‚0.8629460014657726 % _timestamp‚1.7843219107613072e+09 _step‚106 /loss/FRt2/assignment_loss‚3.099263317684942 2loss/SRqqt1/assignment_loss‚0.6073181989102514 2loss/SRqqt2/assignment_loss‚0.8643061270626303 /loss/FBt1/detection_loss‚0.5342132462672516 /loss/FBt2/detection_loss‚1.3104480918158183 .loss/FRt1/detection_loss‚1.436795358250996 1loss/SRqqt2/detection_loss‚1.2907120330370407 epoch‚3 trainer/global_step‚1299 _runtime‚2164.135788616 0loss/FRt1/assignment_loss‚1.5800156027392338 1loss/FBt1/assignment_loss‚0.12540255261501035 0loss/FBt2/assignment_loss‚0.1536210441312519 %loss/total_loss‚12.50810564298704 /loss/FRt2/detection_loss‚0.6430640690068424ÿF~ÉŸ ‚ /loss/FRt2/detection_loss‚0.6430640690068424 1loss/SRqqt1/detection_loss‚0.8629460014657726 % _timestamp‚1.7843219107613072e+09 _step‚106 /loss/FRt2/assignment_loss‚3.099263317684942 2loss/SRqqt1/assignment_loss‚0.6073181989102514 2loss/SRqqt2/assignment_loss‚0.8643061270626303 /loss/FBt1/detection_loss‚0.5342132462672516 /loss/FBt2/detection_loss‚1.3104480918158183 .loss/FRt1/detection_loss‚1.436795358250996 1loss/SRqqt2/detection_loss‚1.2907120330370407 epoch‚3 trainer/global_step‚1299 _runtime‚2164.135788616 0loss/FRt1/assignment_loss‚1.5800156027392338 1loss/FBt1/assignment_loss‚0.12540255261501035 0loss/FBt2/assignment_loss‚0.1536210441312519 %loss/total_loss‚12.50810564298704j,é:“  ‚(:ˆ ¯êÒȱæ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚268.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïã¯-¡ ‚(:¢ ¯êÒ¹óÇ®# disk.nvme0c0n1.in‚ 3883.87109375( disk.nvme0c0n1.out‚31301.12255859375! disk.nvme0n1.in‚ 3883.87109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚578.4625015258789 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0.046875# disk.dm-0.out‚31301.12255859375$ memory_percent‚94.05996799468994! proc.memory.rssMB‚ 2883.796875 disk.sda.out‚ 141.1484375 disk.rbd0.in‚0 disk.rbd2.out‚ 10.966796875 proc.cpu.threads‚17 network.sent‚ 2.319464e+06 network.recv‚572543 disk.sda3.out‚ 141.1484375 disk.rbd0.out‚0) disk./.usagePercent‚38.82110383976894 disk.dm-1.out‚ 141.1484375& disk.nvme0n1.out‚31301.12255859375 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 3883.87109375 disk.sda1.out‚0( proc.memory.availableMB‚ 973.21484375 disk.sda.in‚ 0.01171875 disk.sda2.in‚0# disk.rbd1.out‚15548.03955078125 disk.sda1.in‚0* proc.memory.percent‚17.601299285888672 cpu‚25.47339629996229 disk.sda2.out‚0€f9”¢ ‚(:‰ ¯êÒõÌâ« gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚238.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07íG%£ ‚(:š ¯êÒ‚ãÞ® disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 141.3671875 proc.cpu.threads‚17) disk./.usagePercent‚38.82111510410044 disk.rbd2.out‚ 10.966796875( disk.nvme0c0n1.out‚31310.26318359375 disk.rbd0.in‚0 disk.dm-0.in‚ 3884.703125 disk.sda.in‚ 0.01171875 disk.sda.out‚ 141.3671875 disk.sda2.out‚0# disk.rbd1.out‚15735.36767578125 disk.rbd0.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 3884.703125& disk.nvme0n1.out‚31310.26318359375 disk.sda2.in‚0 network.recv‚576189 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 973.03515625 network.sent‚ 2.38244e+06 disk.rbd2.in‚0.015625# disk.dm-0.out‚31310.26318359375$ memory_percent‚94.06106472015381 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789 cpu‚25.47300786204276$ disk./.usageGB‚578.4626693725586 disk.dm-1.out‚ 141.3671875 disk.nvme0n1.in‚ 3884.703125 disk.rbd1.in‚0.046875Å`‰?”¤ ‚(:‰ Ÿ¯êÒ¸—«­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚270.781' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wo(¥ ‚(: Ÿ¯êғᯠdisk.sda1.out‚0$ memory_percent‚94.05968189239502# disk.rbd1.out‚15763.44580078125# disk.nvme0c0n1.in‚ 3885.69140625 network.sent‚ 2.387533e+06$ disk./.usageGB‚578.4625434875488 disk.sda.in‚ 0.01171875( proc.memory.availableMB‚ 973.26171875 disk.rbd1.in‚0.046875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 141.6640625! proc.memory.rssMB‚ 2884.265625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 3885.69140625 disk.sda.out‚ 141.6640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3885.69140625 disk.drbd2165.out‚0 network.recv‚579604 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚11.03857421875! disk.dm-0.out‚31322.283203125 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 141.6640625* disk./.usagePercent‚38.821106655851814$ disk.nvme0n1.out‚31322.283203125 disk.sda1.in‚0& disk.nvme0c0n1.out‚31322.283203125) proc.memory.percent‚17.60416030883789 cpu‚25.4721562764919 proc.cpu.threads‚17l<Óo“¦ ‚(:ˆ ®¯êÒ²é« gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.699 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OpИ&§ ‚(:› ®¯êҊ묯# disk./.usageGB‚578.462474822998! disk.nvme0n1.in‚ 3885.69140625 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.sda1.in‚0 disk.sda3.out‚ 141.828125 cpu‚25.47179530331728 network.sent‚ 2.392047e+06' disk.nvme0c0n1.out‚31327.6904296875% disk.nvme0n1.out‚31327.6904296875 disk.sda.out‚ 141.828125 disk.rbd2.in‚0.015625" disk.dm-0.out‚31327.6904296875 disk.drbd2165.out‚0$ memory_percent‚94.06044483184814 disk.dm-0.in‚ 3885.69140625 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 141.828125( proc.memory.availableMB‚ 973.13671875 disk.sda2.out‚0) proc.memory.percent‚17.60416030883789 network.recv‚582646( disk./.usagePercent‚38.8211020477162# disk.nvme0c0n1.in‚ 3885.69140625 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875# disk.rbd1.out‚15949.46142578125! proc.memory.rssMB‚ 2884.265625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚11.12451171875ú¨îò”¨ ‚(:‰ ½¯êÒÀ€Ã­ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚276.209' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ASBB8© ‚(:— ½¯êÒéܹ¯ disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.dm-1.out‚ 142.10546875 disk.sda3.out‚ 142.10546875# disk.rbd1.out‚15959.13330078125 disk.sda1.out‚0! proc.memory.rssMB‚ 2884.265625 cpu‚25.47109478489225 network.recv‚586051 disk.dm-0.out‚ 31340.44531$&}ê25# disk./.usageGB‚578.462345123291" disk.nvme0n1.out‚ 31340.4453125 disk.sda.out‚ 142.10546875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚11.48876953125 disk.dm-0.in‚ 3947.8515625 network.sent‚ 2.397624e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 972.87890625 proc.cpu.threads‚17* disk./.usagePercent‚38.821093343460035 disk.nvme0n1.in‚ 3947.8515625 disk.sda1.in‚0 disk.sda.in‚ 0.01171875 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 3947.8515625$ disk.nvme0c0n1.out‚ 31340.4453125$ memory_percent‚94.06201839447021 disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.60416030883789Ȩ8É2ª - _wandbruntime‚2241  _runtime‚2241ø\¬‹Å« ¿ _runtime‚ 2241.46033864 _step‚107 & lr-AdamW/pg1‚0.0002724841479835842 & lr-AdamW/pg2‚0.0002724841479835842 trainer/global_step‚1349 $ _timestamp‚1.784321988085611e+09!6£Öɬ à $ _timestamp‚1.784321988085611e+09 _runtime‚ 2241.46033864 _step‚107 & lr-AdamW/pg1‚0.0002724841479835842 & lr-AdamW/pg2‚0.0002724841479835842 trainer/global_step‚1349k3iRÓ2­ - _wandbruntime‚2241  _runtime‚2241Ó¯í'® çO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387971169995406 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1636266981376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1267353084416 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11668535 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3826045 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106578973010944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386704615739392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843219880869303e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386872432197632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚1349 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386872406024192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388141130671616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11478489 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11478483 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1636234542080 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388139759108608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104942738468864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1266528828046 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388139759108608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11668535 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11678508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388141130671616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386872432197632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1268698473984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104943106560000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11478489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106579373541376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387969799967116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11678508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1268698473984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386704641167360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2241.461759573 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11478483 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3826275 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386872406024192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7290283TÜŠ3ñ'¯ ëO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388139759108608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104942738468864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1266528828046 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388139759108608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11668535 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11678508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388141130671616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386872432197632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1268698473984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104943106560000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11478489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106579373541376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387969799967116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11678508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1268698473984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386704641167360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2241.461759573 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11478483 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3826275 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386872406024192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7290283 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387971169995406 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1636266981376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1267353084416 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11668535 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3826045 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106578973010944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386704615739392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843219880869303e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386872432197632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚1349 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386872406024192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388141130671616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11478489 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11478483 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1636234542080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1267452294404 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388420056444416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387150432927744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388418684874240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11677013 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11486695 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7295479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106657660662784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7295253 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1637468218368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105020224876544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105020592967680 trainer/global_step‚1349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11677013 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11486689 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11687000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1637435786240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚388248605211538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106658061186048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387150432927744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1266107693970 % _timestamp‚1.7843219894876595e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11486689 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚ 2242.86254392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387150406754304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1269623516672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1268278119936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387150406754304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3828928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386982522945536 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388420056444416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3466551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚109 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3828707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11687000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0“Îز ôM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106658061186048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387150432927744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1266107693970 % _timestamp‚1.7843219894876595e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11486689 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚ 2242.86254392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387150406754304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1269623516672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1268278119936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387150406754304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3828928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386982522945536 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388420056444416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3466551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚109 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3828707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11687000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1267452294404 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388420056444416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387150432927744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388418684874240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11677013 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11486695 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7295479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106657660662784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7295253 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1637468218368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105020224876544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105020592967680 trainer/global_step‚1349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11677013 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11486689 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11687000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1637435786240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚388248605211538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154mÈ*˜I2³ - _wandbruntime‚2242  _runtime‚2242Ï4h’´ ÿ 0loss/FRt1/assignment_loss‚1.5983369776180811 /loss/FRt2/detection_loss‚0.6458674052315686 /loss/FBt1/detection_loss‚0.6660818724274088 % _timestamp‚1.7843219895681617e+09 _step‚110 2loss/SRqqt1/assignment_loss‚0.6378849715071828 2loss/SRqqt2/assignment_loss‚0.9127145196703792 1loss/SRqqt1/detection_loss‚0.7025723034814102 1loss/SRqqt2/detection_loss‚1.2116287187520975 _runtime‚2242.942757176 /loss/FBt2/detection_loss‚1.2895166223932957 $loss/total_loss‚12.4132874984063 epoch‚3 /loss/FRt2/assignment_loss‚3.052679596024577 1loss/FBt1/assignment_loss‚0.07869666621530669 1loss/FBt2/assignment_loss‚0.28812912180986305 /loss/FRt1/detection_loss‚1.3291787232751293 trainer/global_step‚1349Ú¤& µ ƒ /loss/FBt2/detection_loss‚1.2895166223932957 $loss/total_loss‚12.4132874984063 epoch‚3 /loss/FRt2/assignment_loss‚3.052679596024577 1loss/FBt1/assignment_loss‚0.07869666621530669 1loss/FBt2/assignment_loss‚0.28812912180986305 /loss/FRt1/detection_loss‚1.3291787232751293 trainer/global_step‚1349 0loss/FRt1/assignment_loss‚1.5983369776180811 /loss/FRt2/detection_loss‚0.6458674052315686 /loss/FBt1/detection_loss‚0.6660818724274088 % _timestamp‚1.7843219895681617e+09 _step‚110 2loss/SRqqt1/assignment_loss‚0.6378849715071828 2loss/SRqqt2/assignment_loss‚0.9127145196703792 1loss/SRqqt1/detection_loss‚0.7025723034814102 1loss/SRqqt2/detection_loss‚1.2116287187520975 _runtime‚2242.942757176nËݲž¶ ‚(:“ ̯êÒ‰üÕ­ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚334.396' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.87920000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k« · ‚(:• ̯êÒúôó¯ network.recv‚589334) disk./.usagePercent‚38.82109283144497# disk.nvme0c0n1.in‚ 3948.80859375) proc.memory.percent‚17.60416030883789$ disk./.usageGB‚578.4623374938965" disk.nvme0c0n1.out‚ 31349.96875 disk.rbd0.out‚0 disk.nvme0n1.out‚ 31349.96875 disk.sda2.in‚0 cpu‚25.470615435764657 disk.rbd0.in‚0 disk.dm-0.in‚ 3948.80859375 network.sent‚ 2.402181e+06 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875( proc.memory.availableMB‚ 973.05078125 disk.sda3.in‚ 0.01171875 disk.dm-0.out‚ 31349.96875 disk.sda1.out‚0! proc.memory.rssMB‚ 2884.265625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 142.33203125! disk.nvme0n1.in‚ 3948.80859375# disk.rbd1.out‚16203.63330078125 disk.rbd2.in‚0.015625 disk.rbd2.out‚11.48876953125 disk.sda1.in‚0$ memory_percent‚94.06096935272217 disk.dm-1.out‚ 142.33203125 disk.sda3.out‚ 142.33203125 disk.sda2.out‚0 disk.rbd1.in‚0.046875 disk.drbd2165.in‚0[§X”¸ ‚(:‰ Û¯êÒ²¹Ý­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚278.846' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÒf*¹ ‚(:Ÿ Û¯êÒøÁǯ disk.drbd2165.out‚0% disk.nvme0n1.out‚31377.2490234375$ disk./.usageGB‚578.5232772827148" disk.dm-0.out‚31377.2490234375 disk.sda3.out‚ 143.05078125 disk.sda.out‚ 143.05078125 proc.cpu.threads‚17 network.sent‚ 2.465234e+06 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 973.02734375! proc.memory.rssMB‚ 2884.265625 disk.rbd0.in‚0 disk.dm-0.in‚ 3948.8515625 disk.drbd2165.in‚0 disk.dm-1.out‚ 143.05078125' disk.nvme0c0n1.out‚31377.2490234375 disk.sda2.in‚0 cpu‚25.469942837885252$ memory_percent‚94.06111240386963) proc.memory.percent‚17.60416030883789 disk.rbd1.in‚0.046875 disk.rbd2.out‚ 11.607421875 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 network.recv‚593867) disk./.usagePercent‚38.82518255180445# disk.rbd1.out‚16270.71142578125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 3948.8515625 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.01171875 disk.nvme0n1.in‚ 3948.8515625ŸÓ“”º ‚(:‰ ê¯êÒï´¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚134.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚042ã» ‚(:“ ê¯êÒÇîÚ° disk.sda2.out‚0 disk.rbd1.in‚0.046875 cpu‚25.46946135700828 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 145.9609375 proc.cpu.threads‚17 network.sent‚ 2.470284e+06 disk.dm-0.in‚ 3949.6875 disk.sda3.in‚ 0.01171875$ memory_percent‚94.06037330627441) disk./.usagePercent‚38.89305398924457 disk.sda3.out‚ 145.9609375 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 3949.6875 disk.sda2.in‚0! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789 network.recv‚597215 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚32008.21923828125& disk.nvme0n1.out‚32008.21923828125 disk.sda.out‚ 145.9609375 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 11.830078125 disk.sda.in‚ 0.01171875# disk.rbd1.out‚16409.78955078125# disk./.usageGB‚579.534610748291 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875 disk.nvme0n1.in‚ 3949.6875' proc.memory.availableMB‚ 973.1484375 disk.rbd0.in‚0# disk.dm-0.out‚32008.21923828125\¨(ŒŸ¼ ‚(:” ù¯êÒ÷óŸ­ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Ìd1½ ‚(:¦ ù¯êÒ¯ž©®$ disk./.usageGB‚579.8473510742188# disk.dm-0.out‚32099.38427734375 disk.rbd2.in‚0.015625$ memory_percent‚94.06037330627441 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚38.91404225492419 disk.dm-0.in‚ 4055.47265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.sda1.out‚0( disk.nvme0c0n1.out‚32099.38427734375 disk.rbd2.out‚11.95556640625! disk.nvme0n1.in‚ 4055.47265625 cpu‚25.469066684361973& disk.nvme0n1.out‚32099.38427734375) proc.memory.percent‚17.60416030883789 network.sent‚ 2.475475e+06 network.recv‚600468 disk.sda.out‚ 147.89453125 disk.rbd1.in‚0.046875 disk.rbd0.out‚0' proc.memory.availableMB‚ 973.1484375 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 147.89453125 disk.sda1.in‚0 disk.sda3.out‚ 147.89453125 disk.sda2.in‚0! proc.memory.rssMB‚ 2884.265625# disk.nvme0c0n1.in‚ 4055.47265625# disk.rbd1.out‚16417.91455078125u<9ž¾ ‚(:“ ˆ°êÒùÏà® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚275.062' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.01240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ð~+,¿ ‚(:¡ ˆ°êÒº³²¯# disk./.usageGB‚583.633171081543 disk.rbd0.out‚0 disk.rbd1.in‚0.046875 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda.out‚ 149.9453125 disk.sda2.in‚0# disk.rbd1.out‚16622.99267578125 cpu‚25.468491505737283 network.recv‚604064( disk.nvme0c0n1.out‚34298.13330078125 disk.sda2.out‚0$ memory_percent‚94.06192302703857( proc.memory.availableMB‚ 972.89453125 network.sent‚ 2.480513e+06* disk./.usagePercent‚39.168111812130284 disk.sda1.in‚0 disk.dm-1.out‚ 149.9453125& disk.nvme0n1.out‚34298.13330078125 disk.sda.in‚ 0.01171875! proc.memory.rssMB‚ 2884.265625 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 149.9453125 disk.rbd2.out‚11.95556640625 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 4055.7109375 disk.dm-0.in‚ 4055.7109375# disk.dm-0.out‚34234.50830078125" disk.nvme0c0n1.in‚ 4055.7109375) proc.memory.percent‚17.60416030883789Šôñþ2À - _wandbruntime‚2321  _runtime‚2321†ÅQ‰ÇÁ Á _step‚111 & lr-AdamW/pg1‚0.0002723133921229003 & lr-AdamW/pg2‚0.0002723133921229003 trainer/global_step‚1399 % _timestamp‚1.7843220682453225e+09 _runtime‚2321.620022685{ Ù„Ë Å _runtime‚2321.620022685 _step‚111 & lr-AdamW/pg1‚0.0002723133921229003 & lr-AdamW/pg2‚0.0002723133921229003 trainer/global_step‚1399 % _timestamp‚1.7843220682453225e+09oþƒh´2à - _wandbruntime‚2321  _runtime‚2321xf‘í'Ä çO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402086111544832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3963256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1696338904064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12094287 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311421340832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400772442529792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400772442529792 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402087483107840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3591153 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108817426944000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400598704648192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1313669015040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12104260 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12094287 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1312765941154 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401910125989024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108817058852864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400772468703232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7554148 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110513800384512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400598730076160 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401911496017314 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _runtime‚2321.621812394 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3591158 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11888783 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400772468703232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315014404608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚1399 _step‚112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402087483107840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11888783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315014404608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110513397756928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1696373440512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚1399 _step‚112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402087483107840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11888783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315014404608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110513397756928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1696373440512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402086111544832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3963256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1696338904064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12094287 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311421340832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400772442529792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400772442529792 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402087483107840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3591153 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108817426944000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400598704648192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1313669015040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12104260 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12094287 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1312765941154 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401910125989024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108817058852864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648p O2Æ - _wandbruntime‚2323  _runtime‚2323¹î†÷&Ç ñM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11896995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚113 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402366408880640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7559453 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110592484596736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1697536715776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7559706 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1314594050560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1312344807078 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843220696559522e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11896995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1315939447296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1697571245056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12112752 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401050469433344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11896989 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3965757 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108894913351680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401050443259904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚402366408880640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401050469433344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _runtime‚2323.030971314 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11896989 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3965757 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108894913351680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401050443259904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚402366408880640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401050469433344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _runtime‚2323.030971314 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚113 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402366408880640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7559453 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110592484596736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1697536715776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7559706 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1314594050560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1312344807078 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843220696559522e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11896995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1315939447296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1697571245056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12112752 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2q'¡Žô2É - _wandbruntime‚2323  _runtime‚2323Â-̹Ê ÿ .loss/FRt2/detection_loss‚0.634354435191845 1loss/SRqqt1/detection_loss‚0.7451565578204989 _runtime‚2323.106175032 0loss/FRt1/assignment_loss‚1.4813553837646964 2loss/SRqqt2/assignment_loss‚0.8480141173447446 1loss/FBt2/assignment_loss‚0.15289232218550503 .loss/FBt1/detection_loss‚0.645947075973249 /loss/FBt2/detection_loss‚1.1531855147331953 1loss/SRqqt2/detection_loss‚1.2934309797972896 trainer/global_step‚1399 2loss/SRqqt1/assignment_loss‚0.6317260428111754 1loss/FBt1/assignment_loss‚0.10720279062124595 /loss/FRt1/detection_loss‚1.4102804215695133 %loss/total_loss‚12.10619902815171 epoch‚3 % _timestamp‚1.7843220697314844e+09 _step‚114 0loss/FRt2/assignment_loss‚3.0026533863387512Aù™+ Ë ƒ 0loss/FRt1/assignment_loss‚1.4813553837646964 2loss/SRqqt2/assignment_loss‚0.8480141173447446 1loss/FBt2/assignment_loss‚0.15289232218550503 .loss/FBt1/detection_loss‚0.645947075973249 /loss/FBt2/detection_loss‚1.1531855147331953 1loss/SRqqt2/detection_loss‚1.2934309797972896 trainer/global_step‚1399 2loss/SRqqt1/assignment_loss‚0.6317260428111754 1loss/FBt1/assignment_loss‚0.10720279062124595 /loss/FRt1/detection_loss‚1.4102804215695133 %loss/total_loss‚12.10619902815171 epoch‚3 % _timestamp‚1.7843220697314844e+09 _step‚114 0loss/FRt2/assignment_loss‚3.0026533863387512 .loss/FRt2/detection_loss‚0.634354435191845 1loss/SRqqt1/detection_loss‚0.7451565578204989 _runtime‚2323.106175032rÃ8mº)Ì ‚(:ž —°êÒœ•Ý­) disk./.usagePercent‚39.40416560713682 disk.rbd2.in‚0.015625" disk.dm-0.out‚39156.7451171875 disk.sda1.out‚0# memory_percent‚94.0605640411377 cpu‚25.46809196635194 proc.cpu.threads‚17 disk.dm-1.out‚ 152.5546875 disk.sda3.in‚ 0.01171875! disk.nvme0n1.in‚ 4056.54296875 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-0.in‚ 4056.54296875 disk.drbd2165.out‚0 disk.rbd0.out‚0 network.recv‚607436 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4056.54296875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚39156.7451171875% disk.nvme0n1.out‚39156.7451171875! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789 network.sent‚ 2.485581e+06$ disk./.usageGB‚587.1505432128906 disk.sda2.in‚0# disk.rbd1.out‚16647.16455078125' proc.memory.availableMB‚ 973.1171875 disk.rbd1.in‚0.046875 disk.rbd2.out‚12.06591796875 disk.sda3.out‚ 152.5546875 disk.sda.out‚ 152.5546875ãnð”Í ‚(:‰ —°êÒÙŸŸ® gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 (”Î ‚(:‰ ¦°êÒŠöƬ gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚303.818' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.7636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×±Ž)Ï ‚(:ž ¦°êÒü®ž¯# disk./.usageGB‚587.359561920166# disk.dm-0.out‚39210.57763671875 network.sent‚ 2.548655e+06 network.recv‚611956 disk.rbd2.out‚12.28857421875& disk.nvme0n1.out‚39210.57763671875 disk.sda.in‚ 0.01171875$ memory_percent‚94.06061172485352& proc.memory.availableMB‚ 973.109375 cpu‚25.467414825585287) disk./.usagePercent‚39.41819302796045 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.sda.out‚ 155.94140625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 155.94140625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0.046875) proc.memory.percent‚17.60416030883789 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 155.94140625! disk.nvme0c0n1.in‚ 4056.546875# disk.rbd1.out‚16821.07080078125 disk.sda1.in‚0( disk.nvme0c0n1.out‚39210.57763671875 disk.sda2.in‚0 disk.dm-0.in‚ 4056.546875 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 4056.546875! proc.memory.rssMB‚ 2884.265625 proc.cpu.threads‚17FµµŸÐ ‚(:” µ°êÒ»ƒ£¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚300.629' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.125800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•EûÑ ‚(:“ µ°êÒÍ©î¯ disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 4057.515625$ disk./.usageGB‚587.5834922790527' proc.memory.availableMB‚ 973.1015625! proc.memory.rssMB‚ 2884.265625 disk.sda2.in‚0 disk.rbd2.out‚ 12.96484375 disk.dm-0.in‚ 4057.515625 disk.dm-0.out‚39404.28515625 disk.sda3.in‚ 0.01171875% disk.nvme0c0n1.out‚39404.28515625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 156.46484375 cpu‚25.46687096072018 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 156.46484375$ memory_percent‚94.06065940856934 network.recv‚615555 disk.rbd1.in‚0.046875# disk.rbd1.out‚16848.61767578125 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 network.sent‚ 2.553744e+06) disk./.usagePercent‚39.43322118223542 disk.sda.out‚ 156.46484375# disk.nvme0n1.out‚39404.28515625 disk.sda1.in‚0) proc.memory.percent‚17.60416030883789 disk.nvme0n1.in‚ 4057.515625û8˜yÒ ‚(:‰ Ä°êÒ×–‘­ gpu.0.gpu‚66.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0. Ù‚uncorrectedMemoryErrors‚0.Hç¬Ó ‚(: Ä°êÒ±Çî®( disk.nvme0c0n1.out‚40470.10302734375 disk.sda.in‚ 0.01171875 network.sent‚ 2.558894e+06 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 4166.6875 disk.drbd2165.in‚0 disk.sda3.out‚ 156.84375 disk.sda2.in‚0' proc.memory.availableMB‚ 972.8515625$ disk./.usageGB‚587.4430618286133! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789 disk.nvme0c0n1.in‚ 4166.6875 disk.sda.out‚ 156.84375$ memory_percent‚94.06218528747559 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.46657831868294 network.recv‚618750 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚40470.10302734375# disk.rbd1.out‚16950.49267578125 disk.rbd0.in‚0 disk.rbd2.out‚ 12.96484375# disk.dm-0.out‚40470.10302734375 disk.dm-1.out‚ 156.84375 disk.nvme0n1.in‚ 4166.6875) disk./.usagePercent‚39.42379677687743 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17t ‡žÔ ‚(:“ Ó°êÒûÄଠgpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.668' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.53360000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s=eÉÕ ‚(:ƒ Ó°êÒ¤Ž¬® disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 4166.6875 disk.drbd2165.out‚0% disk.nvme0n1.out‚40719.5244140625 disk.drbd2165.in‚0 disk.dm-1.out‚157.5$ memory_percent‚94.06235218048096 network.recv‚622537 disk.sda3.out‚157.5 disk.nvme0n1.in‚ 4166.6875 disk.sda.out‚157.5 disk.sda1.in‚0) disk./.usagePercent‚39.41579014124407$ disk./.usageGB‚587.3237571716309 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚13.13134765625" disk.dm-0.out‚40719.5244140625' disk.nvme0c0n1.out‚40719.5244140625 disk.sda2.in‚0( proc.memory.availableMB‚ 972.82421875) proc.memory.percent‚17.60416030883789 network.sent‚ 2.564041e+06 disk.rbd1.in‚0.046875 cpu‚25.465968021982754 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4166.6875# disk.rbd1.out‚16958.61767578125 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875! proc.memory.rssMB‚ 2884.265625\7ð}•Ö ‚(:Š â°êÒôýï« gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚296.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«©$× ‚(:™ â°êÒ¬Ìϯ disk.sda3.out‚ 157.69921875) proc.memory.percent‚17.60416030883789 cpu‚25.465532069451946 disk.rbd2.out‚13.13134765625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 972.83203125! proc.memory.rssMB‚ 2884.265625 disk.sda.in‚ 0.01171875 disk.sda.out‚ 157.69921875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 4167.64453125 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 40785.359375# disk.rbd1.out‚17068.85205078125 disk.dm-1.out‚ 157.69921875# disk.nvme0c0n1.in‚ 4167.64453125* disk./.usagePercent‚39.415821886178314$ disk./.usageGB‚587.3242301940918! disk.nvme0n1.in‚ 4167.64453125 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875$ memory_percent‚94.06230449676514 network.sent‚ 2.569115e+06 disk.sda2.in‚0 disk.rbd1.in‚0.046875 proc.cpu.threads‚17 network.recv‚625964! disk.nvme0n1.out‚ 40785.359375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.out‚ 40785.359375­c2Ø - _wandbruntime‚2400  _runtime‚2400¢z—ÉÙ Ã ' lr-AdamW/pg2‚0.00027213394746857764 trainer/global_step‚1449 % _timestamp‚1.7843221476228528e+09 _runtime‚2400.997471349 _step‚115 ' lr-AdamW/pg1‚0.00027213394746857764„2KžÍÚ Ç ' lr-AdamW/pg1‚0.00027213394746857764 ' lr-AdamW/pg2‚0.00027213394746857764 trainer/global_step‚1449 % _timestamp‚1.7843221476228528e+09 _runtime‚2400.997471349 _step‚115swr4)2Û - _wandbruntime‚2400  _runtime‚2400l æ‘ì'Ü æO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415851822039222 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112691379236864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101393 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414672479035392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4101129 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416032463981056 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3718303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416033835544064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718308 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114448006610944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1756222649344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1359003054262 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12520039 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414672505208832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1359984945664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414672479035392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416032463981056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114447601886208 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12299089 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7819432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1449 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 $ _timestamp‚1.784322147624011e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12299083 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _runtime‚2400.998846285 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚416033835544064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112691747328000 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1756259282944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414492793556992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12299089 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415850452010932 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ÷ð'Ý êO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414672505208832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1359984945664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414672479035392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416032463981056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114447601886208 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12299089 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7819432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1449 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 $ _timestamp‚1.784322147624011e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12299083 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _runtime‚2400.998846285 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚416033835544064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112691747328000 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1756259282944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414492793556992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12299089 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415850452010932 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415851822039222 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112691379236864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101393 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414672479035392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4101129 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416032463981056 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3718303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416033835544064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718308 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114448006610944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1756222649344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1359003054262 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12520039 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736tKü_Ò2Þ - _wandbruntime‚2402  _runtime‚2402qÂœþß ñM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12538504 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7824955 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7824695 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416130627283756 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114526679132672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12307295 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1757408770560 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12528517 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414770675335168 trainer/global_step‚1449 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚117 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12307295 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1360909981184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4103849 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414770700763136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24835806 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1360909981184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_·‰£ùsplit_bytes.small_pool.allocated‚ 1757445396992 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2402.384898898 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12307289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12528517 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4104104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112768865644544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926kl´û&à õM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416312761316864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414950479765504 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3720846 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843221490099802e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416129257255354 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1359926520620 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1362255377920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12538504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414950505938944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416312761316864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1360909981184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1757445396992 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2402.384898898 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12307289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12528517 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4104104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112768865644544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12538504 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7824955 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7824695 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416130627283756 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114526679132672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12307295 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1757408770560 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12528517 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414770675335168 trainer/global_step‚1449 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚117 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12307295 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1360909981184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4103849 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414770700763136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24835806 E2üg(ë ‚(: ž±êÒÁÍÍ® network.recv‚640483 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.dm-1.out‚ 158.98046875 disk.nvme0c0n1.in‚ 4379.21875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.out‚0 cpu‚25.46356870470926 disk.rbd2.out‚13.23779296875# disk.dm-0.out‚41099.56884765625 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 973.30078125! proc.memory.rssMB‚ 2884.265625 network.sent‚ 2.64735e+06 disk.nvme0n1.in‚ 4379.21875# disk.rbd1.out‚17358.86767578125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚41099.56884765625$ memory_percent‚94.05944347381592$ disk./.usageGB‚587.5648651123047 disk.sda.out‚ 158.98046875 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.60416030883789* disk./.usagePercent‚39.431971097445725& disk.nvme0n1.out‚41099.56884765625 disk.rbd0.out‚0 disk.dm-0.in‚ 4379.21875 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 158.98046875•-Œ=”ì ‚(:‰ ­±êÒ—’ú« gpu.0.gpu‚69.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gê‡!í ‚(:– ­±êÒûŒ°® disk.rbd2.in‚0.015625$ disk./.usageGB‚587.5651741027832 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4379.21875( proc.memory.availableMB‚ 973.05078125 proc.cpu.threads‚17 network.sent‚ 2.652512e+06 disk.rbd2.out‚ 13.30078125 network.recv‚643896& disk.nvme0n1.out‚41103.74462890625 disk.sda.out‚ 159.0859375# disk.rbd1.out‚17400.13330078125# disk.dm-0.out‚41103.74462890625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚41103.74462890625 disk.sda.in‚ 0.01171875 cpu‚25.462892329212355 disk.dm-0.in‚ 4379.21875 disk.rbd1.in‚0.046875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.60416030883789 disk.sda3.out‚ 159.0859375' disk./.usagePercent‚39.431991834056 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 159.0859375$ memory_percent‚94.06096935272217 disk.nvme0n1.in‚ 4379.21875! proc.memory.rssMB‚ 2884.265625âÌè#2î - _wandbruntime‚2479  _runtime‚2479šŠ7]Éï à ' lr-AdamW/pg1‚0.00027194582554601105 ' lr-AdamW/pg2‚0.00027194582554601105 trainer/global_step‚1499 % _timestamp‚1.7843222263329802e+09 _runtime‚2479.707552976 _step‚119Ò§ŽÍð Ç trainer/global_step‚1499 % _timestamp‚1.7843222263329802e+09 _runtime‚2479.707552976 _step‚119 ' lr-AdamW/pg1‚0.00027194582554601105 ' lr-AdamW/pg2‚0.00027194582554601105wz¼Ú2ñ - _wandbruntime‚2479  _runtime‚2479¤# yí'ò çO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1406300876288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚2479.708848542 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428572515540992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12709383 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429978816417280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3845453 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8083184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843222263340352e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429980187980288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407646265856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116565699620864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428572541714432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12709389 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12955764 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429980187980288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12945791 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚117510â’”ó ëO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4237985 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8083443 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3845458 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1815826180096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚1499 ^LDeviceStasõSæ]%tsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1407646265856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1403895567048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118381491264512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429792148061130 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12709383 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429978816417280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3845453 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8083184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843222263340352e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429980187980288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407646265856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116565699620864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428572541714432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12709389 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12955764 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429980187980288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12945791 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1406300876288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚2479.708848542 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428572515540992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12709383 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1408571308544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428850516271104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12717589 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430259113753088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12717595 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12954269 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8088460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚121 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404819033294 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428850542444544 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12717595 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3848001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428664764243968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1406163633728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12964256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428850516271104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843222277198486e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12964256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3847996 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408571308544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428850542444544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8088706 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 5pû&ö õM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428850516271104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843222277198486e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12964256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3847996 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408571308544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428850542444544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8088706 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚430070953305664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428664789671936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1408571308544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428850516271104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12717589 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430259113753088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12717595 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12954269 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8088460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚121 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404819033294 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428850542444544 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12717595 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3848001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428664764243968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1406163633728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12964256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0yóŸeÍ2÷ - _wandbruntime‚2481  _runtime‚2481»2N[ø € /loss/FRt2/detection_loss‚0.6596281553107525 trainer/global_step‚1499 _step‚122 &loss/total_loss‚12.341525394225407 0loss/FRt1/assignment_loss‚1.4708885063345616 2loss/SRqqt2/assignment_loss‚0.9014612838724839 1loss/FBt1/assignment_loss‚0.08743593504633546 1loss/SRqqt1/detection_loss‚0.7236215117013204 1loss/FBt2/assignment_loss‚0.11848290179504092 /loss/FRt1/detection_loss‚1.4295262139411464 /loss/FBt1/detection_loss‚0.8195194434609099 epoch‚3 % _timestamp‚1.7843222278062756e+09 _runtime‚2481.180647214 .loss/FRt2/assignment_loss‚2.91319668636139 2loss/SRqqt1/assignment_loss‚0.5893726851796195 1loss/SRqqt2/detection_loss‚1.2790983643642058 /loss/FBt2/detection_loss‚1.3492937068576398ç h­ ù „ .loss/FRt2/assignment_loss‚2.91319668636139 2loss/SRqqt1/assignment_loss‚0.5893726851796195 1loss/SRqqt2/detection_loss‚1.2790983643642058 /loss/FBt2/detection_loss‚1.3492937068576398 /loss/FRt2/detection_loss‚0.6596281553107525 trainer/global_step‚1499 _step‚122 &loss/total_loss‚12.341525394225407 0loss/FRt1/assignment_loss‚1.4708885063345616 2loss/SRqqt2/assignment_loss‚0.9014612838724839 1loss/FBt1/assignment_loss‚0.08743593504633546 1loss/SRqqt1/detection_loss‚0.7236215117013204 1loss/FBt2/assignment_loss‚0.11848290179504092 /loss/FRt1/detection_loss‚1.4295262139411464 /loss/FBt1/detection_loss‚0.8195194434609099 epoch‚3 % _timestamp‚1.7843222278062756e+09 _runtime‚2481.180647214zi£«2”ú ‚(:‰ ¼±êÒÅò“¬ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Ú%á(û ‚(: ¼±êҶ𣭠network.sent‚ 2.658105e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.046875" disk.dm-0.out‚41126.3603515625 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 160.1640625! proc.memory.rssMB‚ 2884.265625 cpu‚25.46254014386453$ disk./.usageGB‚587.5653228759766 disk.sda.out‚ 160.1640625 disk.rbd0.out‚0 disk.rbd2.out‚ 13.30078125 disk.sda1.in‚0 disk.sda3.out‚ 160.1640625 disk.dm-0.in‚ 4649.38671875 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚94.06106472015381 disk.sda.in‚ 0.01171875# disk.rbd1.out‚17476.66455078125 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 4649.38671875) disk./.usagePercent‚39.43200181834983' disk.nvme0c0n1.out‚41126.3603515625! disk.nvme0n1.in‚ 4649.38671875% disk.nvme0n1.out‚41126.3603515625 disk.rbd0.in‚0( proc.memory.availableMB‚ 973.03515625 network.recv‚646912 disk.drbd2165.out‚0) proc.memory.percent‚17.60416030883789ÏàÉ-;ü ‚(:‰ ˱êÒì­ò« gpu.0.gpu‚32.0 gpu.0.memo¡y¿kYry‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚110.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z×9%ý ‚(:š ˱êÒÂÒç®$ disk./.usageGB‚587.5653495788574 cpu‚25.462236100710378 network.recv‚651596# disk.dm-0.out‚41136.03955078125 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.60416030883789 proc.cpu.threads‚17 disk.dm-0.in‚ 4650.21875 disk.sda1.out‚0 disk.dm-1.out‚ 161.45703125( disk.nvme0c0n1.out‚41136.03955078125& disk.nvme0n1.out‚41136.03955078125 network.sent‚ 2.72068e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 4650.21875 disk.sda.in‚ 0.01171875 disk.rbd1.in‚0.046875$ memory_percent‚94.05961036682129 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 161.45703125 disk.sda.out‚ 161.45703125# disk.rbd1.out‚17519.68017578125' proc.memory.availableMB‚ 973.2734375! proc.memory.rssMB‚ 2884.265625* disk./.usagePercent‚39.432003610402575 disk.rbd0.out‚0 disk.rbd2.out‚ 13.30078125 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.nvme0c0n1.in‚ 4650.21875Lù5»žþ ‚(:“ Ú±êÒŽõ¼­ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚292.063' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.41259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zx õ%ÿ ‚(:š Ú±êÒ„•˜¯ disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 161.92578125 disk.sda3.in‚ 0.01171875$ memory_percent‚94.05913352966309$ disk./.usageGB‚587.5652618408203' proc.memory.availableMB‚ 973.3515625 network.recv‚654198) disk./.usagePercent‚39.43199772222928 disk.drbd2165.out‚0& disk.nvme0n1.out‚41144.21826171875 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 4650.21875 disk.sda3.out‚ 161.92578125 disk.nvme0c0n1.in‚ 4650.21875! proc.memory.rssMB‚ 2884.265625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0.046875 disk.rbd0.out‚0# disk.dm-0.out‚41144.21826171875# disk.rbd1.out‚17582.08642578125 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.461664611579533 disk.nvme0n1.in‚ 4650.21875) proc.memory.percent‚17.60416030883789 network.sent‚ 2.726092e+06( disk.nvme0c0n1.out‚41144.21826171875 disk.sda.out‚ 161.92578125 disk.rbd2.out‚ 13.30078125S"øjŸ€ ‚(:” é±êÒù€Ž¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚276.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.285000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° ¡P+ ‚(:  é±êÒœ—‡® disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.461131512958538 disk.rbd2.out‚ 13.30078125 disk.dm-1.out‚ 162.1953125) disk./.usagePercent‚39.43200463443271 disk.rbd2.in‚0.015625 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 4651.17578125! proc.memory.rssMB‚ 2884.265625) proc.memory.percent‚17.60416030883789# disk.rbd1.out‚17597.27392578125! disk.nvme0n1.in‚ 4651.17578125$ memory_percent‚94.05922889709473 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875' proc.memory.availableMB‚ 973.3359375 network.sent‚ 2.729865e+06$ disk./.usageGB‚587.5653648376465 disk.dm-0.in‚ 4651.17578125 disk.sda1.out‚0 disk.sda3.out‚ 162.1953125 network.recv‚657774 disk.rbd1.in‚0.046875( disk.nvme0c0n1.out‚41152.62841796875 disk.sda.out‚ 162.1953125# disk.dm-0.out‚41152.62841796875& disk.nvme0n1.out‚41152.62841796875 proc.cpu.threads‚17’uüÅž‚ ‚(:“ ø±êÒ¹©‚¬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚336.817' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.36340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lv蛃 ‚(:Œ ø±êÒ®õÒ¯ disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda.in‚ 0.01171875) proc.memory.percent‚17.60416030883789 proc.cpu.threads‚17 disk.sda3.out‚ 162.4609375 disk.sda2.out‚0% proc.memory.availableMB‚ 973.53125 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875$ memory_percent‚94.05803680419922 disk.nvme0c0n1.in‚4850.25& disk.nvme0n1.out‚41157.67529296875 disk.sda.out‚ 162.4609375 cpu‚25.460839773671545 disk.dm-0.in‚4850.25 disk.nvme0n1.in‚4850.25 network.recv‚661211 disk.dm-1.out‚ 162.4609375( disk.nvme0c0n1.out‚41157.67529296875 disk.sda2.in‚0! proc.memory.rssMB‚ 2884.265625 network.sent‚ 2.735064e+06# disk.dm-0.out‚41157.67529296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0.046875# disk.rbd1.out‚17630.47705078125) disk./.usagePercent‚39.43200207435737$ disk./.usageGB‚587.5653266906738 disk.rbd0.in‚0 disk.rbd2.out‚ 13.30078125«o62„ - _wandbruntime‚2558  _runtime‚2558ÀûDÃ… ½ $ lr-AdamW/pg2‚0.00027174903843792 trainer/global_step‚1549 % _timestamp‚1.7843223054953084e+09 _runtime‚2558.869925791 _step‚123 $ lr-AdamW/pg1‚0.00027174903843792¤Nj!dž Á % _timestamp‚1.7843223054953084e+09 _runtime‚2558.869925791 _step‚123 $ lr-AdamW/pg1‚0.00027174903843792 $ lr-AdamW/pg2‚0.00027174903843792 trainer/global_step‚1549{ߟ|2‡ - _wandbruntime‚2558  _runtime‚2558üËþ#ì'ˆ æO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442280996802560 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1453962196480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442472552046592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122316654442496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442472578220032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3972603 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚1549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443926540416512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443925168853504 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8348404 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442472552046592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4376042 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443732474083038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784322305496428e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13381516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13371543 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13119683 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13119689 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1451477280478 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442472578220032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876266346496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13119683 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1876233907200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442280971374592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443925168853504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1450132680156 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443731104054748 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13371543 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443926540416512 _step‚124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122316253912064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1453962196480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13119689 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3972608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2558.871331656 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13381516 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1452616806912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1452616806912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4375801 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995YÊçßð'‰ êO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13119683 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13119689 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1451477280478 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442472578220032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876266346496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13119683 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1876233907200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442280971374592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443925168853504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1450132680156 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443731104054748 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13371543 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443926540416512 _step‚124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122316253912064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1453962196480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13119689 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3972608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2558.871331656 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13381516 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1452616806912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1452616806912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4375801 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442280996802560 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1453962196480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442472552046592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122316654442496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442472578220032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3972603 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚1549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443926540416512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443925168853504 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8348404 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442472552046592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4376042 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443732474083038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784322305496428e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13381516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13371543 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0|ýcàÁ2Š - _wandbruntime‚2560  _runtime‚2560¸]ï"Ô‹ ïM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚444011279327572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120517506412544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1454887239168 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4378533 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 # _timestamp‚1.78432230707754e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4378765 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13390008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1453541842432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122394944218112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13127895 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚125 T@DeviceStatsMonitor.o)¸q!n_train_batch_end/active_bytes.all.allocated‚444205466189312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442750552776704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3975151 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1877470237696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1451056146402 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8353679 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442750578950144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352ET«ùù&Œ óM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13390008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1453541842432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122394944218112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13127895 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚125 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444205466189312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442750552776704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3975151 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1877470237696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1451056146402 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8353679 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442750578950144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚444011279327572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120517506412544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1454887239168 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4378533 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 # _timestamp‚1.78432230707754e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4378765 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498932737536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4513786 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4513538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4099753 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1497714393586 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456372614725632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457871521289728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8613544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1936345834496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456372588552192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456372588552192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457671430076656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456175060283392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457871521289728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457672800104946 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13529983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457872892852736 trainer/global_step‚1599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124314340388864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1936311298048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4099758 VHDeviceStatsMonitor.on_train_batch_start/iÈ£óŠnactive_split_bytes.all.current‚ 402627584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456372614725632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13797295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126251054314496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 2637.49062548 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736ÂÑñ'Ÿ ëO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1936311298048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4099758 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456372614725632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13797295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126251054314496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 2637.49062548 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13529983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498932737536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4513786 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4513538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4099753 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1497714393586 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456372614725632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457871521289728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8613544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1936345834496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456372588552192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456372588552192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457671430076656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456175060283392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457871521289728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457672800104946 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13529983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457872892852736 trainer/global_step‚1599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124314340388864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872€ÆÖ¢2  - _wandbruntime‚2638  _runtime‚26386ÃÙ·÷&¡ ñM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1501203169792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13815760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456650615455744 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚2638.874920504 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1497293259510 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13805773 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456650589282304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4516513 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124391826796544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4516281 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1937515474944 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456452967489536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13538195 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126329744891904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499857773056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126329342271488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124392194887680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458150447055360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456650589282304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458151818625536 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8618814 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4102296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457951605349480 % _timestamp‚1.7843223855000694e+09 _step‚129 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499857773056 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13805773 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8618577 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1498637859944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13815760 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501203169792 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13538189 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1937550004224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13538189 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457950235321078 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0„•1Çü&¢ öM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458150447055360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456650589282304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458151818625536 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8618814 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4102296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457951605349480 % _timestamp‚1.7843223855000694e+09 _step‚129 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499857773056 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13805773 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8618577 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1498637859944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13815760 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501203169792 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13538189 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1937550004224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13538189 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457950235321078 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1501203169792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13815760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456650615455744 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚2638.874920504 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1497293259510 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13805773 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456650589282304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4516513 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124391826796544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4516281 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1937515474944 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456452967489536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13538195 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126329744891904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499857773056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126329342271488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124392194887680²¡;]2£ - _wandbruntime‚2638  _runtime‚2638ižÕp ¤ ƒ 2loss/FBt2/assignment_loss‚0.040217350184878924 /loss/FRt1/detection_loss‚1.3783899850384331 1loss/SRqqt2/detection_loss‚1.1874902409608248 /loss/FBt2/detection_loss‚1.0564167071133852 &loss/total_loss‚11.218176572904888 % _timestamp‚1.7843223855866988e+09 1loss/FBt1/assignment_loss‚0.09995666590993416 1loss/SRqqt1/detection_loss‚0.6960004959202766 epoch‚3 trainer/global_step‚1599 0loss/FRt1/assignment_loss‚1.4380126966238023 0loss/FRt2/assignment_loss‚2.8704621265404415 1loss/SRqqt1/assignment_loss‚0.659549905526419 /loss/FRt2/detection_loss‚0.5862679055795607 0loss/FBt1/detection_loss‚0.47473000297218865 _runtime‚2638.961135226 _step‚130 2loss/SRqqt2/assignment_loss‚0.7306824905347445‡wÍí¥ ˆ 1loss/FBt1/assignment_loss‚0.09995666590993416 1loss/SRqqt1/detection_loss‚0.6960004959202766 epoch‚3 trainer/global_step‚1599 0loss/FRt1/assignment_loss‚1.4380126966238023 0loss/FRt2/assignment_loss‚2.8704621265404415 1loss/SRqqt1/assignment_loss‚0.659549905526419 /loss/FRt2/detection_loss‚0.5862679055795607 0loss/FBt1/detection_loss‚0.47473000297218865 _runtime‚2638.961135226 _step‚130 2loss/SRqqt2/assignment_loss‚0.730682490534Ù@qP!7445 2loss/FBt2/assignment_loss‚0.040217350184878924 /loss/FRt1/detection_loss‚1.3783899850384331 1loss/SRqqt2/detection_loss‚1.1874902409608248 /loss/FBt2/detection_loss‚1.0564167071133852 &loss/total_loss‚11.218176572904888 % _timestamp‚1.7843223855866988e+09‚‚ˆd®’¦ ‚(:‡ Ò²êÒõÌÊ« gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚323.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ûÙg'§ ‚(:œ Ò²êÒ¶ž¯$ disk./.usageGB‚582.4333038330078! disk.dm-0.out‚41219.736328125 disk.nvme0n1.in‚ 5118.890625 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 network.recv‚682548 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 5118.890625 disk.sda2.out‚0$ memory_percent‚94.06101703643799 proc.cpu.threads‚17 disk.sda1.out‚0$ disk.nvme0n1.out‚41219.736328125 disk.rbd1.in‚0.046875 disk.rbd2.out‚13.91552734375( proc.memory.availableMB‚ 973.04296875) disk./.usagePercent‚39.08758771432541 disk.dm-1.out‚ 166.46484375 disk.sda3.in‚ 0.01171875& disk.nvme0c0n1.out‚41219.736328125 disk.sda.out‚ 166.46484375# disk.rbd1.out‚17742.22705078125 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-0.in‚ 5118.890625 disk.rbd0.out‚0 cpu‚25.458232613930313 network.sent‚ 2.823782e+06 disk.sda3.out‚ 166.46484375# proc.memory.rssMB‚ 2884.26953125( proc.memory.percent‚17.6041841506958WDta•¨ ‚(:Š á²êÒå §® gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚330.914' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÅ_*© ‚(:Ÿ á²êÒ ø² disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875( proc.memory.percent‚17.6041841506958# disk.nvme0c0n1.in‚ 5119.72265625& disk.nvme0n1.out‚41228.20458984375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚41228.20458984375 disk.sda.out‚ 166.72265625 disk.sda2.in‚0# disk.dm-0.out‚41228.20458984375 disk.sda1.in‚0 disk.sda.in‚ 0.01171875 disk.rbd1.in‚0.046875# disk.rbd1.out‚17742.22705078125 disk.sda3.out‚ 166.72265625" proc.memory.availableMB‚973.25# proc.memory.rssMB‚ 2884.26953125 network.sent‚ 2.887397e+06$ disk./.usageGB‚582.4334373474121 cpu‚25.47342612020693 proc.cpu.threads‚17 network.recv‚686984 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 13.958984375) disk./.usagePercent‚39.08759667458911! disk.nvme0n1.in‚ 5119.72265625 disk.dm-1.out‚ 166.72265625$ memory_percent‚94.05975341796875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 5119.72265625Ïê0K2ª - _wandbruntime‚2665  _runtime‚2665 ?iäÉ=« Ã{ =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6441264152526855 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.644992470741272 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.440914511680603 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9853982329368591 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4904167056083679 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.009931244887411594 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20441706478595734 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6486490964889526 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061687953770160675 *jet/accuracy_1_of_1‚0.2588103413581848 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5777716636657715 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1383015662431717 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13729678094387054 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9958109259605408 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9708789587020874 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 +jet/accuracy_2_of_3‚0.44768404960632324 trainer/global_step‚1599 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8766824007034302 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09671733528375626 +particle/sensitivity‚0.5935280919075012 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚2665.017983301 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7266062498092651 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02719324827194214 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.07168283313512802 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07460957765579224 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49835771322250366 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07375209778547287 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9333094358444214 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4171208441257477 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6880474090576172 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6639735102653503 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05972091853618622 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.047241274267435074 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9963096380233765 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6807878613471985 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6079336404800415 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9902163743972778 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3585098683834076 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5857786536216736 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6626942753791809 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6078810691833496 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7176303863525391 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904595613479614 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6427628397941589 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_2‚1 epoch‚3 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795052409172058 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5024368762969971 /particle/accuracy_3_of_4‚0.9925724267959595 (particle/accuracy‚0.8355811238288879 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955806732177734 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 )jet/accuracy_2_of_4‚0.738719642162323 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6735385656356812 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5477782487869263 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5370896458625793 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7886545062065125 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6035983562469482 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2991589605808258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 % _timestamp‚1.7843224116426678e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18755820393562317 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7235449552536011 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061687953770160675 *loss/val_total_loss‚12.237431514124742 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5642864108085632 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07640055567026138 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.047786515206098557 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07090375572443008 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14850236475467682 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7460885047912598 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05226827785372734 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.30749785900115967 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07622325420379639 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7577503323554993 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 +jet/accuracy_2_of_2‚0.11006476730108261 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07363711297512054 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918708801269531 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9653838276863098 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07061740756034851 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7149659991264343 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9826918244361877 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6297031044960022 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024916562251746655 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4494187533855438 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9086920619010925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00864266324788332 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.546875 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3643452823162079 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.07835056632757187 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05733242630958557 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5604960918426514 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5509625673294067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7897269129753113 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.29054802656173706 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9958109259605408 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914908409118652 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6757205128669739 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.551611065864563 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.668785810470581 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.051883772015571594 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06191252917051315 particle/accuracy_1_of_1‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979218065738678 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.09520726650953293 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7729195356369019 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9943263530731201 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08811567723751068 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12799744307994843 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8112267255783081 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8487498760223389 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5958234071731567 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.058861181139945984 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9931196570396423 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06349898874759674 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7656840085983276 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6922041177749634 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )jet/accuracy_1_of_3‚0.918333888053894 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +particle/specificity‚0.9017713665962219 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9861790537834167 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6132769584655762 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7415549755096436 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7629913091659546 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8194850087165833 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931890368461609 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019357403740286827 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8112267255783081 +jet/accuracy_3_of_3‚0.05022836849093437 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.13480480015277863 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0167926587164402 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848358631134033 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05972091853618622 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0046538556925952435 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7744307518005371 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07883809506893158 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3266748785972595 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04900201037526131 7validation_average_jet_accuracy‚0.31171730160713196 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22573362290859222 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6179027557373047 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5884731411933899 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3545490801334381 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8104690313339233 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04666556790471077 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6179027557373047 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _step‚131 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6030818223953247 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05742911994457245 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933÷"45¬ È{ particle/accuracy_1_of_2‚1 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9943263530731201 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08811567723751068 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12799744307994843 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8112267255783081 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8487498760223389 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5958234071731567 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.058861181139945984 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9931196570396423 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06349898874759674 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7656840085983276 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6922041177749634 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )jet/accuracy_1_of_3‚0.918333888053894 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +particle/specificity‚0.9017713665962219 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9861790537834167 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6132769584655762 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7415549755096436 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7629913091659546 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8194850087165833 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931890368461609 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019357403740286827 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8112267255783081 +jet/accuracy_3_of_3‚0.05022836849093437 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.13480480015277863 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0167926587164402 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848358631134033 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05972091853618622 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0046538556925952435 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7744307518005371 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07883809506893158 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3266748785972595 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04900201037526131 7validation_average_jet_accuracy‚0.31171730160713196 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22573362290859222 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6179027557373047 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5884731411933899 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3545490801334381 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8104690313339233 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04666556790471077 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6179027557373047 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _step‚131 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6030818223953247 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05742911994457245 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6441264152526855 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.644992470741272 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.440914511680603 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9853982329368591 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4904167056083679 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.009931244887411594 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20441706478595734 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6486490964889526 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061687953770160675 *jet/accuracy_1_of_1‚0.2588103413581848 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5777716636657715 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1383015662431717 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13729678094387054 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9958109259605408 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9708789587020874 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 +jet/accuracy_2_of_3‚0.44768404960632324 trainer/global_step‚1599 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8766824007034302 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09671733528375626 +particle/sensitivity‚0.5935280919075012 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚2665.017983301 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7266062498092651 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02719324827194214 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.07168283313512802 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07460957765579224 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49835771322250366 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07375209778547287 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9333094358444214 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4171208441257477 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6880474090576172 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6639735102653503 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05972091853618622 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.047241274267435074 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9963096380233765 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6807878613471985 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6079336404800415 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9902163743972778 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3585098683834076 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5857786536216736 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6626942753791809 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6078810691833496 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7176303863525391 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904595613479614 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6427628397941589 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_2‚1 epoch‚3 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795052409172058 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5024368762969971 /particle/accuracy_3_of_4‚0.9925724267959595 (particle/accuracy‚0.8355811238288879 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955806732177734 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 )jet/accuracy_2_of_4‚0.738719642162323 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6735385656356812 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5477782487869263 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5370896458625793 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7886545062065125 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6035983562469482 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2991589605808258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 % _timestamp‚1.7843224116426678e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18755820393562317 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7235449552536011 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061687953770160675 *loss/val_total_loss‚12.237431514124742 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5642864108085632 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07640055567026138 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.047786515206098557 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07090375572443008 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14850236475467682 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7460885047912598 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05226827785372734 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.30749785900115967 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07622325420379639 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7577503323554993 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 +jet/accuracy_2_of_2‚0.11006476730108261 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07363711297512054 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918708801269531 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9653838276863098 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07061740756034851 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7149659991264343 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9826918244361877 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6297031044960022 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024916562251746655 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4494187533855438 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9086920619010925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.008642663247d#ÚÛš88332 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.546875 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3643452823162079 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.07835056632757187 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05733242630958557 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5604960918426514 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5509625673294067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7897269129753113 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.29054802656173706 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9958109259605408 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914908409118652 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6757205128669739 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.551611065864563 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.668785810470581 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.051883772015571594 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06191252917051315 particle/accuracy_1_of_1‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979218065738678 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.09520726650953293 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7729195356369019ƒ¸çב­ ‚(:† ð²êÒ­«ˆ¬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚32.0 gpu.0.powerWatts‚92.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4502 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t//O® ‚(:‹ ð²êÒÈÞ¦® disk.drbd2165.out‚0 disk.sda3.out‚ 167.26171875 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚690449" disk.nvme0c0n1.in‚ 5120.6796875 disk.dm-0.out‚ 41641.78125 disk.sda2.out‚0 network.sent‚ 2.892079e+06 disk.nvme0n1.out‚ 41656.90625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0$ memory_percent‚94.04349327087402) disk./.usagePercent‚39.15521466457032 disk.rbd1.in‚0.046875# disk.rbd1.out‚17742.30517578125 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 5120.6796875 disk.sda.out‚ 167.26171875 disk.rbd2.in‚0.015625 disk.sda1.in‚0 cpu‚25.480422023667774$ disk./.usageGB‚583.4409942626953 disk.rbd2.out‚ 13.958984375 disk.dm-1.out‚ 167.26171875) proc.memory.percent‚23.52142333984375 disk.rbd0.out‚0 disk.dm-0.in‚ 5120.6796875 disk.dm-1.in‚ 0.01171875' proc.memory.availableMB‚ 975.9140625" disk.nvme0c0n1.out‚ 41656.90625 proc.memory.rssMB‚3853.75˜eS”¯ ‚(:‰ ÿ²êÒ³šÿ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚288.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öi"í)° ‚(:ž ÿ²êÒž•÷® network.sent‚ 2.949818e+06 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 5315.3046875 disk.sda.out‚ 167.93359375 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚43184.763671875 cpu‚25.477381522914886 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 5315.3046875$ disk.nvme0n1.out‚43184.763671875 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2884.26953125 disk.sda.in‚ 0.01171875 disk.sda2.in‚0( proc.memory.percent‚17.6041841506958 disk.dm-0.in‚ 5315.3046875! disk.dm-0.out‚43184.763671875# memory_percent‚88.1324052810669) proc.memory.availableMB‚ 1944.38671875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 167.93359375 disk.rbd0.out‚0 network.recv‚695246 disk.dm-1.in‚ 0.01171875 disk.sda2.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚39.087456638467884$ disk./.usageGB‚582.4313507080078 disk.dm-1.out‚ 167.93359375# disk.rbd1.out‚17742.30517578125 disk.rbd2.out‚ 14.025390625›‰¢Nž± ‚(:“ Ž³êÒ±£…­ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.304' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.86079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R:p&² ‚(:› Ž³êÒ¥¨’®) disk./.usagePercent‚39.08746431869391 disk.sda2.out‚0 disk.rbd1.in‚0.046875 disk.dm-0.in‚ 5316.13671875( proc.memory.percent‚17.6041841506958$ disk./.usageGB‚582.4314651489258# disk.nvme0c0n1.in‚ 5316.13671875 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875! disk.nvme0n1.in‚ 5316.13671875 disk.rbd2.out‚ 14.025390625 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 disk.sda1.in‚0 disk.dm-1.out‚ 168.140625" memory_percent‚88.132643699646 disk.sda3.out‚ 168.140625 disk.sda.in‚ 0.01171875 disk.sda2.in‚0# proc.memory.rssMB‚ 2884.26953125 cpu‚25.47669535677868 network.sent‚ 2.955384e+06 network.recv‚698398' disk.nvme0c0n1.out‚43196.6591796875" disk.dm-0.out‚43196.6591796875% disk.nvme0n1.out‚43196.6591796875 disk.sda.out‚ 168.140625 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.rbd1.out‚17742.36767578125 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1944.34765625o#ùd”³ ‚(:‰ ³êÒ¹âó« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚110.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ™—3´ ‚(:¨ ³êҋϳ² disk.sda1.in‚0 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 cpu‚25.47628469702066 network.sent‚ 2.960007e+06 disk.sda2.out‚0 disk.rbd2.out‚14.63134765625$ memory_percent‚88.13271522521973 proc.cpu.threads‚17& disk.nvme0n1.out‚43200.77197265625* disk./.usagePercent‚39.087461758618566 disk.sda3.out‚ 168.41015625( disk.nvme0c0n1.out‚43200.77197265625! disk.nvme0n1.in‚ 5316.13671875# disk.rbd1.out‚17742.36767578125# disk.dm-0.out‚43200.77197265625( proc.memory.percent‚17.6041841506958 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0( proc.memory.availableMB‚ 1944.3359375# proc.memory.rssMB‚ 2884.26953125 disk.drbd2165.in‚0 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 5316.13671875 disk.sda2.in‚0 disk.rbd1.in‚0.046875 network.recv‚701879 disk.dm-1.out‚ 168.41015625 disk.sda.out‚ 168.41015625 disk.dm-0.in‚ 5316.13671875$ disk./.usageGB‚582.4314270019531 disk.sda1.out‚0Í žµ ‚(:“ ¬³êҧɮ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.448' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.48959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jIú"¶ ‚(:— ¬³êÒ²µ–¯# proc.memory.rssMB‚ 2884.26953125) disk./.usagePercent‚39.08745535843021 disk.rbd0.out‚0) proc.memory.availableMB‚ 1944.45703125 disk.dm-0.in‚ 5317.09375! disk.dm-0.out‚43210.388671875 network.sent‚ 2.965591e+06 network.recv‚705313$ disk./.usageGB‚582.4313316345215 disk.sda3.out‚ 168.8203125 disk.rbd1.in‚0.046875 disk.sda3.in‚ 0.01171875 disk.dm-1.out‚ 168.8203125$ memory_percent‚88.13197612762451 cpu‚25.475749535605388 disk.sda.out‚ 168.8203125 disk.nvme0c0n1.in‚ 5317.09375& disk.nvme0c0n1.out‚43210.388671875 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0$ disk.nvme0n1.out‚43210.388671875 disk.sda2.out‚0 disk.rbd0.in‚0( proc.memory.percent‚17.6041841506958 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.nvme0n1.in‚ 5317.09375# disk.rbd1.out‚17742.44580078125 disk.rbd2.out‚14.85400390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875 disk.drbd2165.out‚0Y“â'· ‚(:œ »³êÒ׆î­ disk.rbd0.in‚0 disk.sda2.out‚0# disk.rbd1.out‚17742.44580078125 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 169.28515625 cpu‚25.475364680903905$ disk.nvme0n1.out‚43218.259765625 disk.rbd2.out‚15.03564453125 network.recv‚708884& proc.memory.availableMB‚ 1944.65625 disk.sda.out‚ 169.28515625 disk.rbd1.in‚0.046875! disk.dm-0.out‚43218.259765625 disk.sda1.in‚0# proc.memory.rssMB‚ 2884.26953125) disk./.usagePercent‚39.08746969485213$ disk./.usageGB‚582.4315452575684 disk.dm-1.in‚ 0.01171875# memory_percent‚88.1307601928711 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 169.28515625" disk.nvme0c0n1.in‚ 5661.7265625( proc.memory.percent‚17.6041841506958 disk.nvme0n1.in‚ 5661.7265625 disk.rbd2.in‚0.015625 network.sent‚ 2.970161e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 5661.7265625& disk.nvme0c0n1.out‚43218.259765625Â:n°”¸ ‚(:‰ »³êҠĺ® gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚342.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.4972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n;Î}2¹ - _wandbruntime‚2747  _runtime‚2747[6…ÖǺ Á & lr-AdamW/pg1‚0.0002713295197779746 & lr-AdamW/pg2‚0.0002713295197779746 trainer/global_step‚1649 % _timestamp‚1.7843224944017282e+09 _runtime‚2747.776391642 _step‚132tIàÌ» Æ & lr-AdamW/pg1‚0.0002713295197779746 & lr-AdamW/pg2‚0.0002713295197779746 trainer/global_step‚1649 % _timestamp‚1.7843224944017282e+09 _runtime‚2747.776391642 _step‚132„ò¢e¢2¼ - _wandbruntime‚2747  _runtime‚2747~Mãóí'½ çO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475653069338112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128237141110784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14282034 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚2747.777699962 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4235826 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1551091282432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1551091282432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14292007 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475651697775104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128236773019648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14072295 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843224944029152e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475444864190904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475651697775104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14072289 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚475443494162614 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1552436672000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474100632666112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14072289 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4235821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474100632666112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474100606492672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8921681 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚473895083552768 _step‚133 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14282034 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8921925 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8921925 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14282034 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚2747.777699962 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4235826 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1551091282432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1551091282432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14292007 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475651697775104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128236773019648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14072295 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843224944029152e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475444864190904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475651697775104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14072289 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚475443494162614 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1552436672000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474100632666112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14072289 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4235821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474100632666112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474100606492672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8921681 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚473895083552768 _step‚133 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14282034 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572…æ³¼2¿ - _wandbruntime‚2749  _runtime‚2749 Ÿ²¥aÀ ñM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474378607222784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2004819976192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1553361714688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843224959730544e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475930623540736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚134 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1552016317952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475931995110912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474172965330944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚475723669435438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14080495 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8927220 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14080501 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14300499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474378633396224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130319479926784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14080501 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4688620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8926984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2004852408320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4238369 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14080495 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475931995110912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130319079403520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549359504060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128314627518464 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚2749.347901422 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14290512 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1550704104494 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1649 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128314259427328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚475722299407036 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175=~a«ü&Á öM _step‚134 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1552016317952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475931995110912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474172965330944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚475723669435438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14080495 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8927220 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14080501 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14300499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474378633396224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130319479926784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14080501 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4688620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8926984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2004852408320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4238369 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14080495 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475931995110912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130319079403520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549359504060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128314627518464 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚2749.347901422 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14290512 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1550704104494 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1649 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128314259427328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚475722299407036 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474378607222784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2004819976192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1553361714688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843224959730544e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475930623540736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648†•^-2 - _wandbruntime‚2749  _runtime‚2749%²—^à ÿ 0loss/FRt1/assignment_loss‚1.2482041075029446 .loss/FRt2/detection_loss‚0.666910937660709 1loss/SRqqt1/detection_loss‚0.7392999713503734 _step‚135 0loss/FRt2/assignment_loss‚2.7804720779820546 /loss/FRt1/detection_loss‚1.3839150887837577 epoch‚4 trainer/global_step‚1649 2loss/SRqqt1/assignment_loss‚0.5460563708206708 2loss/SRqqt2/assignment_loss‚0.7306229407542176 1loss/FBt1/assignment_loss‚0.06235476030017072 % _timestamp‚1.7843224960600193e+09 _runtime‚2749.434364791 1loss/FBt2/assignment_loss‚0.18306379581118343 0loss/SRqqt2/detection_loss‚1.164144076223142 .loss/FBt1/detection_loss‚0.667156301344946 /loss/FBt2/detection_loss‚1.4271436147391796 &loss/total_loss‚11.599344043273351E練 Ä „ 1loss/FBt2/assignment_loss‚0.18306379581118343 0loss/SRqqt2/detection_loss‚1.164144076223142 .loss/FBt1/detection_loss‚0.667156301344946 /loss/FBt2/detection_loss‚1.4271436147391796 &loss/total_loss‚11.599344043273351 0loss/FRt1/assignment_loss‚1.2482041075029446 .loss/FRt2/detection_loss‚0.666910937660709 1loss/SRqqt1/detection_loss‚0.7392999713503734 _step‚135 0loss/FRt2/assignment_loss‚2.7804720779820546 /loss/FRt1/detection_loss‚1.3839150887837577 epoch‚4 trainer/global_step‚1649 2loss/SRqqt1/assignment_loss‚0.5460563708206708 2loss/SRqqt2/assignment_loss‚0.7306229407542176 1loss/FBt1/assignment_loss‚0.06235476030017072 % _timestamp‚1.7843224960600193e+09 _runtime‚2749.434364791‡¯|¼žÅ ‚(:“ ʳêÒìÒÒ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚274.025' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.80499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÂ0‰/Æ ‚(:¤ ʳêÒÃþÄ® disk.rbd0.in‚0# disk.dm-0.out‚43223.97119140625$ memory_percent‚88.13066482543945 cpu‚25.47476604325466 network.sent‚ 2.975208e+06 disk.rbd2.out‚15.03564453125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚43223.97119140625 disk.sda.in‚ 0.01171875 disk.sda.out‚ 169.59765625* proc.memory.percent‚17.601442337036133 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.046875 disk.drbd2165.in‚0 disk.dm-1.out‚ 169.59765625 disk.sda3.out‚ 169.59765625 disk.nvme0n1.in‚ 5662.0859375& disk.nvme0n1.out‚43223.97119140625# disk.rbd1.out‚17742.50830078125 network.recv‚712049 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 5662.0859375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875) disk./.usagePercent‚39.08753523278089$ disk./.usageGB‚582.4325218200684 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 5662.0859375' proc.memory.availableMB‚ 1944.671875" proc.memory.rssMB‚ 2883.8203125}”Ç ‚(:‰ Ù³êÒ¼¡©¬ gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚334.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûú"Ð*È ‚(:Ÿ Ù³êÒµ¹Ä¯ disk.rbd1.in‚0.046875 disk.dm-1.out‚ 169.8359375 network.recv‚716782$ disk./.usageGB‚582.4328384399414# disk.dm-0.out‚43232.20458984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0# disk.rbd1.out‚17742.50830078125 disk.sda.out‚ 169.8359375 disk.dm-0.in‚ 5663.046875 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 169.8359375& disk.nvme0n1.out‚43232.20458984375) proc.memory.availableMB‚ 1944.73828125 disk.rbd2.out‚ 15.154296875( disk.nvme0c0n1.out‚43232.20458984375 disk.nvme0n1.in‚ 5663.046875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17$ memory_percent‚88.13025951385498 cpu‚25.474346768791904) disk./.usagePercent‚39.08755648140623 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2883.83203125! disk.nvme0c0n1.in‚ 5663.046875 disk.sda2.in‚0 network.sent‚ 3.03832e+06 disk.sda2.out‚0* proc.memory.percent‚17.601513862609863uO>”É ‚(:‰ è³êÒú˜´« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚273.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äï/-Ê ‚(:¢ è³êÒ¸Þ¡¯) disk./.usagePercent‚39.08755801745144 disk.dm-1.out‚ 169.9921875 disk.sda.out‚ 169.9921875 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda3.out‚ 169.9921875 disk.drbd2165.out‚0 disk.sda.in‚ 0.01171875$ memory_percent‚88.13331127166748) proc.memory.availableMB‚ 1944.23828125 disk.dm-0.in‚ 5663.87890625# disk.dm-0.out‚43243.09912109375( disk.nvme0c0n1.out‚43243.09912109375& disk.nvme0n1.out‚43243.09912109375 disk.sda2.in‚0 network.recv‚719861 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd2.out‚ 15.154296875# proc.memory.rssMB‚ 2884.26953125# disk.rbd1.out‚17742.58642578125 disk.rbd0.in‚0 cpu‚25.47377797953296# disk./.usageGB‚582.432861328125 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 5663.87890625 disk.rbd1.in‚0.046875( proc.memory.percent‚17.6041841506958# disk.nvme0c0n1.in‚ 5663.87890625 disk.sda3.in‚ 0.01171875 network.sent‚ 3.043427e+06 disk.rbd0.out‚0:nDdŸË ‚(:” ÷³êÒßØ£® gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚251.092' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.21840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r›·#Ì ‚(:˜ ÷³êÒ¹†ƒ¯ disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 5807.296875 disk.dm-0.in‚ 5807.296875# disk.dm-0.out‚43247.84228515625 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2884.26953125 network.recv‚723438 disk.rbd0.in‚0 disk.dm-1.out‚ 170.453125( disk.nvme0c0n1.out‚43247.84228515625# disk.rbd1.out‚17742.58642578125 disk.drbd2165.in‚0 disk.sda3.out‚ 170.453125 disk.sda2.out‚0 network.sent‚ 3.049007e+06( disk./.usagePercent‚39.0875569934213 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.rbd2.out‚ 15.169921875 disk.drbd2165.out‚0$ disk./.usageGB‚582.4328460693359 disk.rbd2.in‚0.015625( proc.memory.availableMB‚ 1945.5390625 cpu‚25.473391342041143 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 5807.296875& disk.nvme0n1.out‚43247.84228515625 disk.sda.out‚ 170.453125 disk.rbd0.out‚0 disk.sda1.in‚0# memory_percent‚88.1253719329834( proc.memory.percent‚17.6041841506958âõ2žÍ ‚(:“ †´êÒ´ƒò« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚286.112' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.22240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õœ¾ñ1Î ‚(:¦ †´êÒ©’Í®$ disk./.usageGB‚582.4329605102539 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 5808.25390625 disk.sda1.out‚0& disk.nvme0n1.out‚43259.41552734375 disk.sda2.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚43259.41552734375! disk.nvme0n1.in‚ 5808.25390625 disk.sda.out‚ 170.69140625 network.sent‚ 3.053527e+06 network.recv‚726403 disk.rbd2.out‚ 15.169921875 disk.sda3.out‚ 170.69140625 disk.rbd1.in‚0.046875 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5808.25390625# disk.rbd1.out‚17742.64892578125) disk./.usagePercent‚39.08756467364733 disk.rbd2.in‚0.015625 disk.sda2.in‚0) proc.memory.availableMB‚ 1945.91015625 cpu‚25.472748739605 proc.cpu.threads‚17# disk.dm-0.out‚43259.41552734375 disk.rbd0.in‚0$ memory_percent‚88.12310695648193# proc.memory.rssMB‚ 2883.83203125 disk.dm-1.out‚ 170.69140625 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875* proc.memory.percent‚17.601513862609863š:àð2Ï - _wandbruntime‚2826  _runtime‚2826~ÓÀÆÐ À _step‚136 & lr-AdamW/pg1‚0.0002711068151710198 & lr-AdamW/pg2‚0.0002711068151710198 trainer/global_step‚1699 $ _timestamp‚1.784322573026998e+09 _runtime‚2826.401587289ŽuîÊËÑ Å & lr-AdamW/pg1‚0.0002711068151710198 & lr-AdamW/pg2‚0.0002711068151710198 trainer/global_step‚1699 $ _timestamp‚1.784322573026998e+09 _runtime‚2826.401587289 _step‚136ˆe2Ò - _wandbruntime‚2826  _runtime‚2826í‰ùcÏÓ çO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488000642998272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚1699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1597407213056 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489598050211328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063748638720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488000669171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063781078016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14717759 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489599421774336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648¤%µ2ò'Ô ìO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚2826.402888896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 % _timestamp‚1.7843225730280845e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29200354 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488000642998272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132111461494784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134174842042368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1596017751244 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488000669171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063781078016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14717759 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489599421774336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488000642998272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚1699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1597407213056 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489598050211328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063748638720 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4365514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4827521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489663995457346 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14490801 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9193040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488278669901824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14726251 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596941217602 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1599677645312 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9192794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14726251 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132188947902464 trainer/global_step‚1699 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843225744125993e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4827280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1599677645312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4365519 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14716264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2064979759104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489662625428944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488278669901824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚488067054239744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _step‚138 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14490795 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488067028811776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29207059 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14490801 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29217052 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14490795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488278643728384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134253927661568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489878347547136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489878347547136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064947326976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820V¶·7¤× öM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064947326976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134253527138304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1598332248576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14716264 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4365514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4827521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489663995457346 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14490801 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9193040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488278669901824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14726251 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596941217602 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1599677645312 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9192794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14726251 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132188947902464 trainer/global_step‚1699 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843225744125993e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4827280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1599677645312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4365519 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14716264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2064979759104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489662625428944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488278669901824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚488067054239744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _step‚138 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14490795 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488067028811776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29207059 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14490801 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29217052 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14490795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488278643728384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134253927661568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489878347547136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489878347547136ŠÌ,U`2Ø - _wandbruntime‚2827  _runtime‚2827õ}àÙ ý 2loss/SRqqt1/assignment_loss‚0.5899994347337886 0loss/FBt1/detection_loss‚0.48236763488419854 &loss/total_loss‚10.908186250376929 epoch‚4 trainer/global_step‚1699 /loss/FRt2/detection_loss‚0.6376592235016967 1loss/SRqqt1/detection_loss‚0.6850203016118542 1loss/FBt1/assignment_loss‚0.08497589460721998 /loss/FRt1/detection_loss‚1.3359482625296764 _step‚139 0loss/FRt1/assignment_loss‚1.2452490083102523 2loss/SRqqt2/assignment_loss‚0.8583034035814068 0loss/FBt2/assignment_loss‚0.1334395855020765 0loss/SRqqt2/detection_loss‚1.081899355790627 /loss/FBt2/detection_loss‚1.1002538547452008 # _timestamp‚1.78432257449919e+09 _runtime‚2827.873521882 .loss/FRt2/assignment_loss‚2.67307029057893T–Ú ‚ 1loss/FBt1/assignment_loss‚0.08497589460721998 /loss/FRt1/detection_loss‚1.3359482625296764 _step‚139 0loss/FRt1/assignment_loss‚1.2452490083102523 2loss/SRqqt2/assignment_loss‚0.8583034035814068 0loss/FBt2/assignment_loss‚0.1334395855020765 0loss/SRqqt2/detection_loss‚1.081899355790627 /loss/FBt2/detection_loss‚1.1002538547452008 # _timestamp‚1.78432257449919e+09 _runtime‚2827.873521882 .loss/FRt2/assignment_loss‚2.67307029057893 2loss/SRqqt1/assignment_loss‚0.5899994347337886 0loss/FBt1/detection_loss‚0.48236763488419854 &loss/total_loss‚10.908186250376929 epoch‚4 trainer/global_step‚1699 /loss/FRt2/detection_loss‚0.6376592235016967 1loss/SRqqt1/detection_loss‚0.6850203016118542‹x]6 žÛ ‚(:“ •´êÒÿÕÒ« gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚258.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.77759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G¡é*Ü ‚(:Ÿ •´êÒÓßö­ disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 170.890625 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd2.out‚15.73681640625( disk.nvme0c0n1.out‚43266.84130859375 disk.sda.out‚ 170.890625* proc.memory.percent‚17.601513862609863 proc.cpu.threads‚17 network.sent‚ 3.059162e+06 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚43266.84130859375 network.recv‚730270 disk.sda3.out‚ 170.890625 disk.nvme0n1.in‚ 5809.0859375) disk./.usagePercent‚39.08756723372267 cpu‚25.472467135383333 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.rbd1.in‚0.046875 disk.dm-0.in‚ 5809.0859375 disk.dm-1.in‚ 0.01171875 disk.sda2.in‚0# disk.rbd1.out‚17742.64892578125$ disk./.usageGB‚582.4329986572266 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 5809.0859375 disk.sda.in‚ 0.01171875$ memory_percent‚88.12503814697266& proc.memory.availableMB‚ 1945.59375# proc.memory.rssMB‚ 2883.83203125# disk.dm-0.out‚43266.84130859375ìKú Ý ‚(:… ¤´êÒë“°¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚292.6' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.52 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÁÙ™1Þ ‚(:¦ ¤´êÒÐÏ® disk.rbd2.in‚0.015625 network.recv‚734248" disk.nvme0c0n1.in‚ 5809.0859375 disk.dm-1.in‚ 0.01171875$ memory_percent‚88.12611103057861( proc.memory.percent‚17.6041841506958& disk.nvme0n1.out‚43272.12353515625 disk.sda2.out‚0 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1945.41796875$ disk./.usageGB‚582.4326972961426 disk.rbd2.out‚15.73681640625 disk.dm-1.out‚ 171.26953125 disk.sda3.out‚ 171.26953125# proc.memory.rssMB‚ 2884.26953125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚43272.12353515625# disk.rbd1.out‚17742.64892578125 disk.sda3.in‚ 0.01171875 network.sent‚ 3.122188e+06 disk.sda.in‚ 0.01171875# disk.dm-0.out‚43272.12353515625 disk.sda1.out‚0 disk.nvme0n1.in‚ 5809.0859375 disk.sda2.in‚0 disk.dm-0.in‚ 5809.0859375 disk.sda1.in‚0 cpu‚25.471826374765353 disk.sda.out‚ 171.26953125 disk.rbd0.out‚0 disk.drbd2165.out‚0) disk./.usagePercent‚39.08754700912746´ëÞÖ”ß ‚(:‰ ³´êÒÙ¾¶­ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚255.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ ?£0à ‚(:¥ ³´êҬï disk.sda1.in‚0! disk.nvme0n1.in‚ 5926.28515625 disk.rbd1.in‚0.046875 cpu‚25.471512433272853 disk.rbd0.out‚0# disk.dm-0.out‚43280.51416015625# disk.nvme0c0n1.in‚ 5926.28515625# disk.rbd1.out‚17742.72705078125 network.sent‚ 3.127234e+06 network.recv‚737696 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 171.56640625& disk.nvme0n1.out‚43280.51416015625( proc.memory.percent‚17.5980806350708 disk.rbd0.in‚0 disk.dm-0.in‚ 5926.28515625 disk.sda3.in‚ 0.01171875# proc.memory.rssMB‚ 2883.26953125 proc.cpu.threads‚17 disk.rbd2.out‚ 15.85546875 disk.sda1.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚43280.51416015625 disk.sda2.out‚0( proc.memory.availableMB‚ 1946.1640625 disk.drbd2165.out‚0 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 171.56640625 disk.sda.out‚ 171.56640625$ memory_percent‚88.12155723571777) disk./.usagePercent‚39.08754752114253$ disk./.usageGB‚582.4327049255371ãš,*á ‚(:Ÿ ´êÒ¬¹Ø­* proc.memory.percent‚17.601513862609863* disk./.usagePercent‚39.087576449993904 disk.rbd2.out‚ 15.85546875 disk.sda1.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 5927.1171875 disk.nvme0n1.in‚ 5927.1171875 disk.rbd1.in‚0.046875 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2883.83203125 cpu‚25.47109083596967 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 172.01953125' disk.nvme0c0n1.out‚43289.2275390625 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875% disk.nvme0n1.out‚43289.2275390625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 5927.1171875" disk.dm-0.out‚43289.2275390625 network.recv‚740828$ disk./.usageGB‚582.4331359863281 network.sent‚ 3.131801e+06 disk.drbd2165.in‚0 disk.sda3.out‚ 172.01953125# disk.rbd1.out‚17742.72705078125 disk.sda.out‚ 172.01953125 disk.sda2.in‚0$ memory_percent‚88.12541961669922& proc.memory.availableMB‚ 1945.53125Q9Ç{žâ ‚(:“ ´êÒ½ˆ® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚291.533' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.30660000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ÄPàŸã ‚(:” Ñ´êÒ‡ü¡¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚292.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.462199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­¼å ä ‚(:• Ñ´êÒÔÔ® disk.dm-1.out‚172.25 network.recv‚743402 disk.rbd2.in‚0.015625 disk.sda1.out‚0( disk.nvme0c0n1.out‚43293.88720703125 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2883.83203125 network.sent‚ 3.136142e+06" disk.nvme0c0n1.in‚ 5927.1171875# disk.rbd1.out‚17742.78955078125* proc.memory.percent‚17.601513862609863 disk.sda.in‚ 0.01171875 disk.sda.out‚172.25 disk.sda2.in‚0) proc.memory.availableMB‚ 1945.84765625 disk.rbd2.out‚15.89892578125# disk.dm-0.out‚43293.88720703125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 5927.1171875 disk.rbd0.in‚0 disk.dm-0.in‚ 5927.1171875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0.046875 disk.dm-1.in‚ 0.01171875* disk./.usagePercent‚39.087577218016506$ disk./.usageGB‚582.4331474304199 disk.drbd2165.in‚0& disk.nvme0n1.out‚43293.88720703125# memory_percent‚88.1234884262085 disk.sda3.in‚ 0.01171875 disk.sda3.out‚172.25 cpu‚25.47063111996606 Àpˆ2å - _wandbruntime‚2905  _runtime‚2905ñ‹»OÈæ  _step‚140 ' lr-AdamW/pg1‚0.00027087549926660913 ' lr-AdamW/pg2‚0.00027087549926660913 trainer/global_step‚1749 $ _timestamp‚1.784322651658499e+09 _runtime‚2905.033098391°jHÍç Ç _runtime‚2905.033098391 _step‚140 ' lr-AdamW/pg1‚0.00027087549926660913 ' lr-AdamW/pg2‚0.00027087549926660913 trainer/global_step‚1749 $ _timestamp‚1.784322651658499e+09Œ¢4<ö2è - _wandbruntime‚2905  _runtime‚2905djÝí'é çO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15143511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1642254864352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501683235942400 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚503544402647552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645068533248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4490121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4963044 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645068533248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4490126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503324146206430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚503545774210560 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9453436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501900679503872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9453165 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138108915385344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985413787648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843226516596634e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501900679503872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1643723143680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14892895 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4963310 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123536134144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14892889 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138109318012928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14892895 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚503545774210560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15133538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚1749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1643723143680 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1640910264030 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123501597696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14892889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚141 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚503544402647552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15133538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚503325516234720+Ù8J%#ê ìO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645068533248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4490121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4963044 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645068533248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4490126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503324146206430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚503545774210560 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9453436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501900679503872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9453165 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138108915385344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985413787648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843226516596634e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501900679503872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1643723143680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14892895 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4963310 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123536134144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14892889 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138109318012928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14892895 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚503545774210560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15133538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚1749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1643723143680 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1640910264030 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123501597696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14892889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚141 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚503544402647552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15133538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚503325516234720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15143511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1642254864352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501683235942400 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚503544402647552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚¶]ƒMÍ5 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124689546240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503604321479254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚503823328413184 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9458492 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643178330710 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 _step‚142 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14901095 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136063268286464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚503824699983360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚502178706407424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4966086 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15152003 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138187992361984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124724075520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14901095 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138187589741568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503602951450852 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1641833730276 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1644648179200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136062900195328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501961143148544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15142016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1645993575936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1645993575936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4492664 $ _timestamp‚1.784322653043176e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15142016 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502178706407424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14901101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ƒ û&í õM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9458492 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643178330710 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 _step‚142 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14901095 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136063268286464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚503824699983360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚502178706407424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4966086 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15152003 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138187992361984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124724075520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14901095 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138187589741568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503602951450852 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1641833730276 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1644648179200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136062900195328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501961143148544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15142016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1645993575936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1645993575936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4492664 $ _timestamp‚1.784322653043176e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15142016 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502178706407424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14901101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1644648179200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚503824699983360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4965828 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124689546240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503604321479254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚503823328413184 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ž:Æ®U2î - _wandbruntime‚2906  _runtime‚2906VU°©ï ý 0loss/FRt1/assignment_loss‚1.1693218474485436 1loss/SRqqt2/assignment_loss‚0.790499018227321 /loss/FRt1/detection_loss‚1.4210080940668395 /loss/FRt2/detection_loss‚0.6546223696402398 1loss/SRqqt2/detection_loss‚1.2918497009543322 /loss/FBt2/detection_loss‚1.5580680785781664 trainer/global_step‚1749 $ _timestamp‚1.784322653130213e+09 1loss/FBt1/assignment_loss‚0.06963751634051236 .loss/FBt1/detection_loss‚0.939977373614056 %loss/total_loss‚11.91140692041349 epoch‚4 /loss/FRt2/assignment_loss‚2.638071664138164 1loss/FBt2/assignment_loss‚0.13538220923431293 2loss/SRqqt1/assignment_loss‚0.5316718599339563 1loss/SRqqt1/detection_loss‚0.7112971882370478 _runtime‚2906.504655776 _step‚143Üã¤ð ‚ /loss/FRt1/detection_loss‚1.4210080940668395 /loss/FRt2/detection_loss‚0.6546223696402398 1loss/SRqqt2/detection_loss‚1.2918497009543322 /loss/FBt2/detection_loss‚1.5580680785781664 trainer/global_step‚1749 $ _timestamp‚1.784322653130213e+09 1loss/FBt1/assignment_loss‚0.06963751634051236 .loss/FBt1/detection_loss‚0.939977373614056 %loss/total_loss‚11.91140692041349 epoch‚4 /loss/FRt2/assignment_loss‚2.638071664138164 1loss/FBt2/assignment_loss‚0.13538220923431293 2loss/SRqqt1/assignment_loss‚0.5316718599339563 1loss/SRqqt1/detection_loss‚0.7112971882370478 _runtime‚2906.504655776 _step‚143 0loss/FRt1/assignment_loss‚1.1693218474485436 1loss/SRqqt2/assignment_loss‚0.790499018227321r«¯“ñ ‚(:ˆ à´êÒ–ª® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚340.205' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.041 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t8¤,.ò ‚(:£ à´êÒ¶¾‰¯# disk.rbd1.out‚17742.78955078125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 5928.07421875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 172.51171875 disk.sda.in‚ 0.01171875 cpu‚25.470765025931197 disk.rbd2.out‚16.61767578125 disk.dm-0.in‚ 5928.07421875' disk.nvme0c0n1.out‚43306.5615234375# proc.memory.rssMB‚ 2884.26953125 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 172.51171875 disk.sda2.out‚0( proc.memory.availableMB‚ 1945.1171875" disk.dm-0.out‚43306.5615234375% disk.nvme0n1.out‚43306.5615234375 network.sent‚ 3.141196e+06) disk./.usagePercent‚39.08758592227267# disk./.usageGB‚582.433277130127 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875! disk.nvme0n1.in‚ 5928.07421875 disk.sda.out‚ 172.51171875 disk.sda2.in‚0 disk.drbd2165.in‚0# memory_percent‚88.1279468536377( proc.memory.percent‚17.6041841506958 network.recv‚746434¤Zt ”ó ‚(:‰ ï´êÒ°‘´¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.2116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u*B,ô ‚(:¡ ï´êÒÇšº¯ disk.sda2.out‚0# disk.rbd1.out‚17742.86767578125 disk.sda1.in‚0$ memory_percent‚88.12968730926514) proc.memory.availableMB‚ 1944.83203125 disk.rbd1.in‚0.046875 disk.rbd2.out‚ 16.736328125! disk.nvme0n1.in‚ 5984.63671875& disk.nvme0n1.out‚43311.81005859375 network.recv‚750765 disk.sda.out‚ 172.796875# disk.dm-0.out‚43311.81005859375 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 5984.63671875# disk.nvme0c0n1.in‚ 5984.63671875( disk.nvme0c0n1.out‚43311.81005859375 cpu‚25.471839166575755$ disk./.usageGB‚582.4340057373047 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 172.796875# proc.memory.rssMB‚ 2884.26953125 network.sent‚ 3.204251e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 172.796875) disk./.usagePercent‚39.08763481971172 disk.sda.in‚ 0.01171875( proc.memory.percent‚17.6041841506958 proc.cpu.threads‚17g=¨(õ ‚(: þ´êÒöõ® disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 6077.58203125 disk.rbd1.in‚0.046875# proc.memory.rssMB‚ 2884.26953125( proc.memory.percent‚17.6041841506958) disk./.usagePercent‚39.08763609974938$ memory_percent‚88.12973499298096 cpu‚25.473256166828072 disk.rbd0.in‚0 disk.dm-1.out‚ 173.203125 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 network.recv‚753785 disk.dm-0.in‚ 6077.58203125 disk.sda1.out‚0' disk.nvme0c0n1.out‚43325.4189453125 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 16.736328125 disk.sda3.out‚ 173.203125 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 6077.58203125# disk.rbd1.out‚17742.86767578125 network.sent‚ 3.209304e+06# disk./.usageGB‚582.434024810791 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875% disk.nvme0n1.out‚43325.4189453125 disk.sda.out‚ 173.203125 disk.sda2.in‚0) proc.memory.availableMB‚ 1944.82421875 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17" disk.dm-0.out‚43325.4189453125 disk.sda1.in‚0ý ‹”ö ‚(:‰ þ´êÒúæÒ® gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚48.0 gpu.0.powerWatts‚323.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.7458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä&Z¼Ÿ÷ ‚(:” µêÒ††Ý« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.129' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.025800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Ž§&ø ‚(:› µêÒƨή disk.dm-0.out‚43335.11328125 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 173.51953125 disk.nvme0n1.in‚ 6078.5390625 disk.rbd2.in‚0.015625 network.sent‚ 3.214356e+06) disk./.usagePercent‚39.08755673741377 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 6078.5390625 disk.sda.out‚ 173.51953125 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚582.4328422546387 disk.rbd0.out‚0 disk.rbd2.out‚ 16.8037109375( proc.memory.availableMB‚ 1944.8203125 network.recv‚757590 disk.rbd1.in‚0.046875 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚88.12975883483887 cpu‚25.474318594060634 disk.dm-1.out‚ 173.51953125# disk.nvme0n1.out‚43335.11328125 disk.sda2.in‚0# disk.rbd1.out‚17742.93017578125( proc.memory.percent‚17.6041841506958 proc.cpu.threads‚17% disk.nvme0c0n1.out‚43335.11328125 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2884.26953125 disk.dm-0.in‚ 6078.5390625Þ¨À”ù ‚(:‰ œµêÒ«¥ú« gpu.0.gpu‚80.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚111.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ýÆË&ú ‚(:› œµêÒž“­® disk.drbd2165.in‚0 network.sent‚ 3.219985e+06 disk.drbd2165.out‚0( proc.memory.percent‚17.6041841506958" disk.nvme0c0n1.in‚ 6078.5390625 disk.sda.in‚ 0.01171875 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625 network.recv‚760826 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚43355.56298828125 disk.rbd2.out‚ 16.8037109375 disk.dm-1.in‚ 0.01171875) disk./.usagePercent‚39.08808769703978 disk.sda.out‚ 175.59375 disk.rbd0.out‚0# disk.dm-0.out‚43355.56298828125 disk.sda1.out‚0 disk.sda3.out‚ 175.59375# disk.rbd1.out‚17742.93017578125 disk.rbd0.in‚0$ memory_percent‚88.12978267669678 disk.dm-0.in‚ 6078.5390625) proc.memory.availableMB‚ 1944.81640625# proc.memory.rssMB‚ 2884.26953125 cpu‚25.47545373044694 disk.dm-1.out‚ 175.59375$ disk./.usageGB‚582.4407539367676 disk.nvme0n1.in‚ 6078.5390625 disk.sda1.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚43355.56298828125 disk.sda2.in‚0 disk.sda2.out‚0QÛ2û - _wandbruntime‚2984  _runtime‚2984Û4MÇü Á _step‚144 & lr-AdamW/pg1‚0.0002706355869217309 & lr-AdamW/pg2‚0.0002706355869217309 trainer/global_step‚1799 % _timestamp‚1.7843227306827872e+09 _runtime‚2984.057564611³³ÏªÌý Æ % _timestamp‚1.7843227306827872e+09 _runtime‚2984.057564611 _step‚144 & lr-AdamW/pg1‚0.0002706355869217309 & lr-AdamW/pg2‚0.0002706355869217309 trainer/global_step‚1799™Žú2þ - _wandbruntime‚2984  _runtime‚2984ºnëX†ÿ åO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142043015319552 % _timestamp‚1.7843227306865237e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15569263 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1690039074304 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9719101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15303195 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15303189 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517264472228338 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1691384463872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5102105 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139859734171648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15559290 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚517490755083776 =7DeviceStatsMonitor.on_train_bam^u´e tch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517490755083776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691384463872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1688491977460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517492126646784 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182913056768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617276 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15303189 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5101830 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515800742182912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139860102262784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1687147377138 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1690039074304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517265842256628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15559290 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2182876423168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚515577350279168 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517490755083776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691384463872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1688491977460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517492126646784 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182913056768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617276 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15303189 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5101830 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515800742182912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139860102262784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1687147377138 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1690039074304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517265842256628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15559290 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2182876423168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚515577350279168 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30879163 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2184051794944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15577755 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15567768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚515855206629376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4619814 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516078742913024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5104498 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚515855232057344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5104766 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15311401 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30879163 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2184051794944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15577755 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15567768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚515855206629376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4619814 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516078742913024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5104498 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚515855232057344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5104766 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15311401 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15995015 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15985042 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4744426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531437107520000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143734422646784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531204798250246 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734729090568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15713495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529700778688512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1736355004928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9984646 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145976440604160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529700752515072 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15995015 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15985042 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4744426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531437107520000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143734422646784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531204798250246 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734729090568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15713495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529700778688512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1736355004928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9984646 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145976440604160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529700752515072 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16003507 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146054706188800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143811540963328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529978779418624 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9989649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734307956492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16003507 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15993520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531717404855808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531484973523070 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15721701 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1738625437184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2243165225472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15721695 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1737280040448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4746964 trainer/global_step‚1849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15993520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚529749295538176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843228115720112e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚150 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15721701 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4746969 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146055110906368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143811909054464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚3064.946883661 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5242685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529978779418624 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9989649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734307956492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16003507 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15993520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531717404855808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531484973523070 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15721701 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1738625437184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2243165225472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15721695 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1737280040448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4746964 trainer/global_step‚1849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15993520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚529749295538176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843228115720112e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚150 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15721701 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4746969 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146055110906368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143811909054464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚3064.946883661 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5242685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16003507 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146054706188800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143811540963328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434–¾!±Ä2œ - _wandbruntime‚3065  _runtime‚3065¥Å9 € % _timestamp‚1.7843228116553783e+09 _step‚151 2loss/SRqqt1/assignment_loss‚0.6072644434275334 /loss/FRt1/detection_loss‚1.5064707365888421 &loss/total_loss‚10.454366119622875 epoch‚4 /loss/FRt2/assignment_loss‚2.368359626157611 1loss/FBt1/assignment_loss‚0.06905232603484757 1loss/SRqqt1/detection_loss‚0.6623827604158801 _runtime‚3065.029814596 0loss/FRt1/assignment_loss‚0.9766264173292345 2loss/SRqqt2/assignment_loss‚0.7402910515744934 1loss/FBt2/assignment_loss‚0.14841575684271624 /loss/FRt2/detection_loss‚0.6091387482645337 /loss/FBt1/detection_loss‚0.6117019968013719 trainer/global_step‚1849 1loss/SRqqt2/detection_loss‚1.0193826101025787 .loss/FBt2/detection_loss‚1.135279646083232©ïÀ ž … _runtime‚3065.029814596 0loss/FRt1/assignment_loss‚0.9766264173292345 2loss/SRqqt2/assignment_loss‚0.7402910515744934 1loss/FBt2/assignment_loss‚0.14841575684271624 /loss/FRt2/detection_loss‚0.6091387482645337 /loss/FBt1/detection_loss‚0.6117019968013719 trainer/global_step‚1849 1loss/SRqqt2/detection_loss‚1.0193826101025787 .loss/FBt2/detection_loss‚1.135279646083232 % _timestamp‚1.7843228116553783e+09 _step‚151 2loss/SRqqt1/assignment_loss‚0.6072644434275334 /loss/FRt1/detection_loss‚1.5064707365888421 &loss/total_loss‚10.454366119622875 epoch‚4 /loss/FRt2/assignment_loss‚2.368359626157611 1loss/FBt1/assignment_loss‚0.06905232603484757 1loss/SRqqt1/detection_loss‚0.6623827604158801—G”Ÿ ‚(:‰ …¶êÒõ¯‰¬ gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚334.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò‹uÁ(  ‚(: …¶êÒ­ÊÓ®) disk./.usagePercent‚39.09124683001231# disk.rbd1.out‚17743.27392578125) proc.memory.percent‚17.60420799255371 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1944.84765625 network.recv‚786895 disk.rbd0.in‚0 disk.rbd2.in‚0.015625" disk.dm-0.out‚44250.7900390625 disk.sda.in‚ 0.01171875 proc.cpu.threads‚17$ disk./.usageGB‚582.4878273010254 disk.rbd0.out‚0 disk.rbd2.out‚ 18.376953125 disk.dm-0.in‚ 6256.75390625 disk.sda3.out‚ 182.640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 182.640625 disk.rbd1.in‚0.046875# memory_percent‚88.1295919418335 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875# disk.nvme0c0n1.in‚ 6256.75390625% disk.nvme0n1.out‚44250.7900390625 cpu‚25.483941587544077 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 6256.75390625" proc.memory.rssMB‚ 2884.2734375 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚44250.7900390625 disk.sda.out‚ 182.640625 network.sent‚ 3.349891e+06ä…Șž¡ ‚(:“ ”¶êÒú¢È­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚305.534' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚61.10679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÿâ=¢ ‚(:” ”¶êÒ£¾¬¯) proc.memory.availableMB‚ 1945.06640625 network.recv‚791203 disk.dm-0.out‚ 44258.265625 disk.drbd2165.out‚0! disk.nvme0n1.out‚ 44258.265625 disk.rbd1.in‚0.046875 cpu‚25.484890213761023* disk./.usagePercent‚39.091244525944504 disk.rbd0.in‚0 disk.rbd2.out‚ 18.607421875 disk.sda3.in‚ 0.01171875# disk.rbd1.out‚17743.27392578125 network.sent‚ 3.412909e+06 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 6257.08203125# disk.nvme0c0n1.out‚ 44258.265625! disk.nvme0n1.in‚ 6257.08203125" proc.memory.rssMB‚ 2884.2734375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.out‚ 182.9453125 disk.sda.out‚ 182.9453125 disk.rbd2.in‚0.015625 disk.sda3.out‚ 182.9453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6257.08203125 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚88.12825679779053) proc.memory.percent‚17.60420799255371" disk./.usageGB‚582.48779296875 disk.sda1.in‚0 disk.sda.in‚ 0.01171875^ûqΑ£ ‚(:† £¶êÒÍþ†¬ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚337.25' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.45 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸‘r’/¤ ‚(:¤ £¶êÒ™Æò® disk.sda1.out‚0$ disk./.usageGB‚582.4727439880371" disk.nvme0c0n1.in‚ 6257.7109375& disk.nvme0n1.out‚44292.12255859375 disk.rbd2.in‚0.015625) proc.memory.availableMB‚ 1945.35546875 network.recv‚794824* disk./.usagePercent‚39.090234576221924 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚18.80908203125 disk.dm-0.in‚ 6257.7109375 disk.sda1.in‚0 disk.nvme0n1.in‚ 6257.7109375 disk.sda.out‚ 184.9609375 disk.sda2.out‚0 disk.dm-1.out‚ 184.9609375 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 184.9609375$ memory_percent‚88.12649250030518 cpu‚25.48616828481944 disk.rbd1.in‚0.046875 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 network.sent‚ 3.418096e+06" proc.memory.rssMB‚ 2883.8203125* proc.memory.percent‚17.601442337036133 proc.cpu.threads‚17# disk.rbd1.out‚17743.35205078125 disk.drbd2165.in‚0# disk.dm-0.out‚44292.12255859375( disk.nvme0c0n1.out‚44292.12255859375 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0ù¢»”¥ ‚(:‰ ²¶êÒ˜–­ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚250.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oóff¦ ‚(:‘ ²¶êҨ͙¯ network.recv‚798025 disk.rbd2.in‚0.015625 disk.rbd2.out‚18.80908203125 disk.sda1.in‚0 disk.sda3.out‚ 188.01953125! disk.nvme0c0n1.in‚ 6343.046875" proc.memory.rssMB‚ 2883.8359375* disk./.usagePercent‚39.089737153582895 disk.rbd1.in‚0.046875 disk.dm-0.out‚ 44310.984375 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875" disk./.usageGB‚582.46533203125 disk.sda.out‚ 188.01953125 disk.sda2.in‚0 disk.dm-1.out‚ 188.01953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚88.12723159790039* proc.memory.percent‚17.601537704467773 disk.rbd0.in‚0 disk.nvme0n1.in‚ 6343.046875! disk.nvme0n1.out‚ 44310.984375 disk.sda2.out‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 1945.234375# disk.rbd1.out‚17743.35205078125 network.sent‚ 3.42314e+06 cpu‚25.487262398273494 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 6343.046875# disk.nvme0c0n1.out‚ 44310.984375 dLi”§ ‚(:‰ Á¶êÒ¨¤ô« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚277.566' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ر|Ũ ‚(:” Á¶êÒ³Ôà®& proc.memory.availableMB‚ 1945.21875 network.sent‚ 3.428228e+06$ disk./.usageGB‚582.4657440185547 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625 disk.'º´FZrbd2.out‚ 18.880859375! disk.nvme0c0n1.in‚ 6343.046875$ memory_percent‚88.12732696533203 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚44319.853515625 disk.dm-0.in‚ 6343.046875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 188.578125# disk.rbd1.out‚17743.41455078125 disk.sda3.in‚ 0.01171875 disk.sda.out‚ 188.578125 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.601537704467773 cpu‚25.488358881126672* disk./.usagePercent‚39.089764802396594! disk.dm-0.out‚44319.853515625 disk.sda1.in‚0 disk.sda3.out‚ 188.578125$ disk.nvme0n1.out‚44319.853515625 network.recv‚801600 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875" proc.memory.rssMB‚ 2883.8359375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 6343.046875 disk.sda.in‚ 0.01171875 óÄ2© - _wandbruntime‚3143  _runtime‚3143Žüè>ƪ À % _timestamp‚1.7843228902376168e+09 _runtime‚ 3143.61234057 _step‚152 & lr-AdamW/pg1‚0.0002701300350982019 & lr-AdamW/pg2‚0.0002701300350982019 trainer/global_step‚1899­3 Ë« Å trainer/global_step‚1899 % _timestamp‚1.7843228902376168e+09 _runtime‚ 3143.61234057 _step‚152 & lr-AdamW/pg1‚0.0002701300350982019 & lr-AdamW/pg2‚0.0002701300350982019˜X2¬ - _wandbruntime‚3143  _runtime‚3143:œOï'­ éO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149909812737536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147608374939648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16410794 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543600815194112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843228902388003e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545383459956224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16420767 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5378381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545384831519232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16123795 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4871571 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2301069706752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545145124272154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780966203676 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545384831519232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10249952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5378646 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2301035170304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚543365502668800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16123789 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16123795 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1782670935552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543600789020672ÂgÈô'® îO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784016325120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149909410109952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16420767 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16420767 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5378381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545384831519232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16123795 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4871571 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2301069706752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545145124272154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780966203676 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545384831519232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10249952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5378646 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2301035170304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚543365502668800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16123789 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16123795 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1782670935552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543600789020672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149909812737536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147608374939648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16410794 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543600815194112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843228902388003e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545383459956224 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149988085752320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1783595971072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543878815924224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545663757292032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1784941367808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147686229438464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10255536 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545662385721856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚3145.013618198 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543878789750784 $ _timestamp‚1.784322891638708e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚543643384446976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚154 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5381417 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16429259 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16429259 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4874114 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5381158 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16419272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16132001 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1783595971072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147685861347328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2302224404992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545425299544978 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10255272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543878789750784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16132001 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1784941367808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543878815924224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302258934272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781889670034 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16131995 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149988488372736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16419272©—J^± ÷M E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545662385721856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚3145.013618198 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543878789750784 $ _timestamp‚1.784322891638708e+09 ?5DeviceStatsMo˘CõŸ#nitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚543643384446976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚154 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5381417 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16429259 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16429259 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4874114 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5381158 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16419272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16132001 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1783595971072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147685861347328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2302224404992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545425299544978 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10255272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543878789750784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16132001 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1784941367808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543878815924224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302258934272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781889670034 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16131995 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149988488372736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16419272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545663757292032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545423929516576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4874119 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1780545069600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149988085752320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1783595971072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543878815924224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545663757292032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1784941367808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147686229438464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10255536 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627šûqrÚ2² - _wandbruntime‚3145  _runtime‚3145Ôè³ þ $ _timestamp‚1.784322891720688e+09 _runtime‚ 3145.09529611 .loss/FBt2/detection_loss‚1.477304684993049 &loss/total_loss‚10.705863836793654 _step‚155 1loss/FBt2/assignment_loss‚0.03629563503899259 /loss/FRt2/detection_loss‚0.5994259729733286 1loss/SRqqt1/detection_loss‚0.6762204218995119 1loss/SRqqt2/detection_loss‚1.1038388410158324 /loss/FRt2/assignment_loss‚2.458337386487313 2loss/SRqqt1/assignment_loss‚0.5935777095835175 1loss/FBt1/assignment_loss‚0.02352553495119842 /loss/FBt1/detection_loss‚0.6383837911738222 0loss/FRt1/assignment_loss‚0.9594991889866915 2loss/SRqqt2/assignment_loss‚0.5978756383747647 /loss/FRt1/detection_loss‚1.5415790313156321 epoch‚4 trainer/global_step‚1899¨ûã ´ ƒ &loss/total_loss‚10.705863836793654 _step‚155 1loss/FBt2/assignment_loss‚0.03629563503899259 /loss/FRt2/detection_loss‚0.5994259729733286 1loss/SRqqt1/detection_loss‚0.6762204218995119 1loss/SRqqt2/detection_loss‚1.1038388410158324 /loss/FRt2/assignment_loss‚2.458337386487313 2loss/SRqqt1/assignment_loss‚0.5935777095835175 1loss/FBt1/assignment_loss‚0.02352553495119842 /loss/FBt1/detection_loss‚0.6383837911738222 0loss/FRt1/assignment_loss‚0.9594991889866915 2loss/SRqqt2/assignment_loss‚0.5978756383747647 /loss/FRt1/detection_loss‚1.5415790313156321 epoch‚4 trainer/global_step‚1899 $ _timestamp‚1.784322891720688e+09 _runtime‚ 3145.09529611 .loss/FBt2/detection_loss‚1.477304684993049› ´é”µ ‚(:‰ жêÒ«¶¸« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚172.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— _]/¶ ‚(:¤ жêÒåØí­# disk.nvme0c0n1.in‚ 6344.00390625 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1944.74609375 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.01171875" proc.memory.rssMB‚ 2884.2734375 disk.sda1.out‚0 network.recv‚804734 disk.rbd1.in‚0.046875 network.sent‚ 3.433794e+06& disk.nvme0n1.out‚44331.91943359375 disk.sda2.in‚0# disk.dm-0.out‚44331.91943359375) proc.memory.percent‚17.60420799255371$ disk./.usageGB‚582.4655838012695 disk.dm-1.out‚ 189.0390625 disk.sda3.out‚ 189.0390625( disk.nvme0c0n1.out‚44331.91943359375! disk.nvme0n1.in‚ 6344.00390625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚88.13021183013916) disk./.usagePercent‚39.08975405008015 disk.sda3.in‚ 0.01171875 disk.sda.out‚ 189.0390625 disk.sda2.out‚0 disk.rbd2.out‚ 18.880859375 disk.dm-0.in‚ 6344.00390625 cpu‚25.488114043796738 proc.cpu.threads‚17# disk.rbd1.out‚17743.41455078125 disk.rbd0.out‚0úà Ÿ· ‚(:” ߶êÒñØå­ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.575' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.114999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Êbr,¸ ‚(:¡ ߶êÒ¹¹® disk.rbd0.out‚0 disk.rbd2.out‚18.94384765625) proc.memory.availableMB‚ 1944.95703125 disk.sda2.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2883.8203125* proc.memory.percent‚17.601442337036133& disk.nvme0c0n1.out‚44341.283203125 disk.drbd2165.in‚0 disk.sda3.in‚ 0.01171875$ disk.nvme0n1.out‚44341.283203125 disk.rbd1.in‚0.046875# disk.rbd1.out‚17743.49267578125! disk.dm-0.out‚44341.283203125# disk./.usageGB‚582.468994140625 disk.sda3.out‚ 190.17578125 disk.rbd0.in‚0$ memory_percent‚88.12892436981201 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 190.17578125 network.sent‚ 3.496802e+06" disk.nvme0c0n1.in‚ 6344.8359375 disk.dm-0.in‚ 6344.8359375 cpu‚25.487533027012663 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚809109* disk./.usagePercent‚39.089982920815764 disk.nvme0n1.in‚ 6344.8359375 disk.sda.out‚ 190.17578125 disk.sda2.in‚0 <^Ÿ¹ ‚(:” î¶êÒäÞÉ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.035' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.407000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01U|.º ‚(:£ î¶êÒ„Òâ® disk.sda.in‚ 0.01171875 disk.rbd1.in‚0.046875# disk.rbd1.out‚17743.49267578125 disk.drbd2165.in‚0' proc.memory.availableMB‚ 1944.953125# proc.memory.rssMB‚ 2883.83984375 network.sent‚ 3.501366e+06 disk.sda3.out‚ 192.1640625' disk.nvme0c0n1.out‚44369.3076171875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 6438.97265625" disk.dm-0.out‚44369.3076171875 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 6438.97265625 disk.dm-1.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚18.94384765625$ disk./.usageGB‚582.4690170288086 disk.sda2.in‚0 disk.drbd2165.out‚0$ memory_percent‚88.12894821166992* proc.memory.percent‚17.601561546325684 disk.sda1.out‚0) disk./.usagePercent‚39.08998445686097! disk.nvme0n1.in‚ 6438.97265625% disk.nvme0n1.out‚44369.3076171875 disk.sda.out‚ 192.1640625 disk.sda1.in‚0 cpu‚25.486962026954643 proc.cpu.threads‚17 network.recv‚812481 disk.dm-1.out‚ 192.1640625`«æ6”» ‚(:‰ ý¶êÒÂ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰SM3/¼ ‚(:¤ ý¶êÒ‡à㯠disk.dm-0.in‚ 6439.9296875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 195.57421875( disk.nvme0c0n1.out‚44384.47998046875# disk.rbd1.out‚17743.55517578125 disk.sda1.in‚0 disk.rbd0.out‚0 cpu‚25.486695716838 network.recv‚815944 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 19.0068359375# proc.memory.rssMB‚ 2883.83984375* proc.memory.percent‚17.601561546325684 proc.cpu.threads‚17# disk.dm-0.out‚44384.47998046875 disk.drbd2165.in‚0 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 195.57421875& disk.nvme0n1.out‚44384.47998046875 disk.sda.in‚ 0.01171875 disk.sda.out‚ 195.57421875) proc.memory.availableMB‚ 1944.85546875" disk.nvme0c0n1.in‚ 6439.9296875 network.sent‚ 3.506429e+06$ memory_percent‚88.12954425811768 disk.dm-1.in‚ 0.01171875 disk.nvme0n1.in‚ 6439.9296875) disk./.usagePercent‚39.09016135806713$ disk./.usageGB‚582.4716529846191II«,”½ ‚(:‰ Œ·êÒñØí¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅxÏ0¾ ‚(:¥ Œ·êÒèôº¯ disk.rbd0.out‚0 disk.rbd2.out‚ 19.5419921875 disk.sda3.out‚ 197.75390625) proc.memory.percent‚17.60420799255371 cpu‚25.48606698991145* disk./.usagePercent‚39.089856965108915" disk.dm-0.out‚44410.8603515625# disk.nvme0c0n1.in‚ 6440.76171875' disk.nvme0c0n1.out‚44410.8603515625 proc.cpu.threads‚17 network.sent‚ 3.511633e+06 network.recv‚819109$ disk./.usageGB‚582.4671173095703 disk.sda2.in‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2884.2734375 disk.rbd1.in‚0.046875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875$ memory_percent‚88.13107013702393 disk.sda.out‚ 197.75390625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6440.76171875 disk.sda.in‚ 0.01171875 disk.sda2.out‚0# disk.rbd1.out‚17743.55517578125 disk.dm-0.in‚ 6440.76171875 disk.dm-1.out‚ 197.75390625 disk.dm-1.in‚ 0.01171875% disk.nvme0n1.out‚44410.8603515625 disk.rbd2.in‚0.015625 disk.sda1.in‚0) proc.memory.availableMB‚ 1944.60546875G@2¿ - _wandbruntime‚3222  _runtime‚3222Ï53ÆÀ À ' lr-AdamW/pg1‚0.00026986442809020004 ' lr-AdamW/pg2‚0.00026986442809020004 trainer/global_step‚1949 $ _timestamp‚1.784322969144592e+09 _runtime‚ 3222.5194736 _step‚1560—2,ËÁ Å _runtime‚ 3222.5194736 _step‚156 ' lr-AdamW/pg1‚0.00026986442809020004 ' lr-AdamW/pg2‚0.00026986442809020004 trainer/global_step‚1949 $ _timestamp‚1.784322969144592e+09œcêÖC2 - _wandbruntime‚3222  _runtime‚3222G?[ï'à éO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚157 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16846519 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16836546 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚3222.520410198 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16534089 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151483063414784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557500825526272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557500825526272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1828986866176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10514777 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559329812392448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151482695323648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1949 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4998721 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1830332255744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16836546 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557500851699712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559329812392448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557259617005568 % _timestamp‚1.7843229691456282e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16534095 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360131822080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16846519 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16534095 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4998726 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5516335 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚3222.520410198 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16534089 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151483063414784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557500825526272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557500825526272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1828986866176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10514777 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559329812392448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151482695323648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1949 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4998721 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1830332255744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16836546 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557500851699712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559329812392448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557259617005568 % _timestamp‚1.7843229691456282e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16534095 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360131822080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16846519 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16534095 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4998726 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5516335 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1831257298432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151560181731328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16855011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153921489484288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559608738158080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2361307752960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559610109728256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10520307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559610109728256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚557778826256384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153921894201856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10520040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1829911901696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559364255538484 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16855011 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557778852429824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828126783142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16542295 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5518776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557778852429824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2361344379392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557537498783744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557537473355776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚1949 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16845024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151560549822464a ÙÍþ&Ç øM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557778852429824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2361344379392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557537498783744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557537473355776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚1949 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16845024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151560549822464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1831257298432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151560181731328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16855011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153921489484288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559608738158080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2361307752960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559610109728256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10520307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559610109728256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚557778826256384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153921894201856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10520040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1829911901696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559364255538484 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16855011 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557778852429824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828126783142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16542295 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5518776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352žÊÔ52È - _wandbruntime‚3223  _runtime‚3223¶†î0 gpu.0.powerWatts‚330.234' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.04679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03¢“õ1Ò ‚(:¦ È·êÒÅø•® disk.dm-1.in‚ 0.01171875! disk.nvme0n1.in‚ 6530.94921875# disk.rbd1.out‚17743.69580078125 disk.rbd2.in‚0.015625 network.sent‚ 3.589795e+06 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 201.15234375# disk.nvme0c0n1.in‚ 6530.94921875" proc.memory.rssMB‚ 2884.2734375* disk./.usagePercent‚39.089486522206876# disk.dm-0.out‚44445.20068359375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 19.7509765625# memory_percent‚88.1310224533081 disk.dm-0.in‚ 6530.94921875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚44445.20068359375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 201.15234375 disk.rbd1.in‚0.046875 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1944.61328125 disk.dm-1.out‚ 201.15234375& disk.nvme0n1.out‚44445.20068359375 disk.sda2.out‚0) proc.memory.percent‚17.60420799255371 cpu‚25.48436872661911 network.recv‚833470# disk./.usageGB‚582.461597442627ÄÁ-ç”Ó ‚(:‰ ×·êÒ¤·°« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¸ª Ô ‚(:• ×·êÒìö’¯ disk.sda1.out‚0$ memory_percent‚88.12978267669678) proc.memory.availableMB‚ 1944.81640625 network.recv‚837066 disk.sda3.out‚ 201.59375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.483819624690454* disk./.usagePercent‚39.089489850304815 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 network.sent‚ 3.594316e+06 disk.sda2.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚44454.0439453125 disk.nvme0n1.in‚ 6531.90625% disk.nvme0n1.out‚44454.0439453125 disk.sda1.in‚0 disk.dm-1.out‚ 201.59375# disk.rbd1.out‚17743.77392578125 disk.dm-0.in‚ 6531.90625 disk.nvme0c0n1.in‚ 6531.90625 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 20.5517578125 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.sda.out‚ 201.59375 disk.sda2.in‚0 disk.rbd1.in‚0.046875" proc.memory.rssMB‚ 2883.8359375* proc.memory.percent‚17.601537704467773$ disk./.usageGB‚582.4616470336914' disk.nvme0c0n1.out‚44454.0439453125l¿þo’Õ ‚(:‡ æ·êÒÖÝù« gpu.0.gpu‚4.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚332.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e³o1Ö ‚(:¦ æ·êÒÙŠ€® disk.sda1.in‚0# disk.nvme0c0n1.in‚ 6752.03515625) disk./.usagePercent‚39.09012218891439 disk.rbd0.in‚0' disk.nvme0c0n1.out‚44474.2939453125! disk.nvme0n1.in‚ 6752.03515625 cpu‚25.483459766326693 disk.rbd1.in‚0.046875% disk.nvme0n1.out‚44474.2939453125 disk.sda.out‚ 203.70703125 disk.drbd2165.out‚0 network.recv‚840391 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 203.70703125$ memory_percent‚88.13149929046631 network.sent‚ 3.599955e+06 disk.dm-0.in‚ 6752.03515625 disk.dm-1.out‚ 203.70703125 disk.sda2.out‚0" proc.memory.rssMB‚ 2883.8359375 proc.cpu.threads‚17" disk.dm-0.out‚44474.2939453125 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875$ disk./.usageGB‚582.4710693359375 disk.rbd2.in‚0.015625 disk.sda2.in‚0# disk.rbd1.out‚17743.77392578125* proc.memory.percent‚17.601537704467773 disk.rbd0.out‚0 disk.rbd2.out‚ 20.5517578125 disk.sda3.in‚ 0.01171875) proc.memory.availableMB‚ 1944.53515625úÞzò2× - _wandbruntime‚3301  _runtime‚3301§})éÉØ Ã _runtime‚3301.286371405 _step‚160 ' lr-AdamW/pg1‚0.00026959028958094294 ' lr-AdamW/pg2‚0.00026959028958094294 trainer/global_step‚1999 % _timestamp‚1.7843230479117653e+09Iç ÎÙ È trainer/global_step‚1999 % _timestamp‚1.7843230479117653e+09 _runtime‚3301.286371405 _step‚160 ' lr-AdamW/pg1‚0.00026959028958094294 ' lr-AdamW/pg2‚0.00026959028958094294 x*6Œ2Ú - _wandbruntime‚3301  _runtime‚3301 }—ï'Û éO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1876648186368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17272271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157776479869440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155357015707648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872095829570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17262298 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚571400862031872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10780468 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 % _timestamp‚1.7843230479128695e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚1999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17272271 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157776077241856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873440429892 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573276164828672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1876648186368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573277536391680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5654331 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573025776315970 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚571400888205312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚573027146344260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944389 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10780202 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚571153705914368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‰¹ˆô'Ü îO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5654331 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573025776315970 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚571400888205312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚573027146344260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944389 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10780202 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚571153705914368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1876648186368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17272271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157776479869440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155357015707648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872095829570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17262298 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚571400862031872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10780468 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 % _timestamp‚1.7843230479128695e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚1999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17272271 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157776077241856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873440429892 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573276164828672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1876648186368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573277536391680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16952601 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10785486 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155434870206464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚1999 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5128419 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155434502115328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16952595 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5128414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17280763 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571431562264576 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843230492979896e+09 _step‚162 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _runtime‚3302.672809904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1877573229056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573556462164480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚571678862761984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1876227832320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inac¾ÿ° tive_split_bytes.all.allocated‚157855145739776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573555090594304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1876227832320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17280763 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚571678888935424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16952595 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5657324 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16952601 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6õŠ^yþ&ß øM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5657072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843230492979896e+09 _step‚162 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _runtime‚3302.672809904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1877573229056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573556462164480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚571678862761984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1876227832320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157855145739776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573555090594304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1876227832320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17280763 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚571678888935424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16952595 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5657324 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16952601 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157854743119360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2420275533312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16952601 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10785486 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155434870206464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚1999 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5128419 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155434502115328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16952595 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5128414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17280763 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571431562264576 P&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 .particle/accuracy_3_of_4‚0.997619092464447 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 trainer/global_step‚1999 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.881232738494873 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5420169830322266 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15798570215702057 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1391916573047638 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17080800235271454 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.129060760140419 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867221117019653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11018378287553787 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.989795982837677 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7800543904304504 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0439470000565052 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7697246074676514 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6823735237121582 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7710772156715393 (jet/accuracy_3_of_3‚0.06414745002985 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791865348816 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 jet/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06292404979467392 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6991283297538757 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3893691897392273 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0581771545112133 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6915150880813599 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868274927139282 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6864567995071411 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0638013556599617 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7200415730476379 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10255154967308044 *loss/val_total_loss‚10.182209173871698 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1197049468755722 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886714220046997 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.51583331823349 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.530046820640564 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5248119831085205 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936407208442688 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09355971962213516 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843230753975291e+09 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2383744865655899 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18188419938087463 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.268325537443161 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7473544478416443 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03438909724354744 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_3_of_3‚0.9994933605194092 jet/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922657012939453 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7462018132209778 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17667332291603088 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5902751684188843 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7561303377151489 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.592757523059845 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7976943254470825 6"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7031210064888 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4034399688243866 +particle/specificity‚0.9129061102867126 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43714410066604614 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7267287969589233 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6473700404167175 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6409134864807129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.66053706407547 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18370914459228516 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948128461837769 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7030205726623535 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7423469424247742 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16531474888324738 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9943934082984924 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45627063512802124 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19310110807418823 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7875979542732239 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚4 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7448541522026062 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7976943254470825 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚3328.772805829 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7137147188186646 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7794456481933594 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955765604972839 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1500745564699173 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9695531129837036 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6304343342781067 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14822617173194885 *jet/accuracy_2_of_2‚0.1384061723947525 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14839135110378265 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25937753915786743 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4221319556236267 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6358199119567871 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29634761810302734 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6911255717277527 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023957621306180954 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6263033747673035 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061458684504032135 *jet/accuracy_2_of_4‚0.7724812626838684 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906672239303589 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06341979652643204 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6validation_average_jet_accuracy‚0.3599169850349426 _step‚164 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6622573733329773 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906860589981079 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6576939821243286 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17080800235271454 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8572064638137817 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7473545074462891 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6070747971534729 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18291275203227997 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_4_of_4‚0.9172014594078064 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9881786108016968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06393878161907196 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 (particle/accuracy‚0.8463349342346191 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7534154653549194 particle/accuracy_2_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6830309629440308 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5890286564826965 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17615433037281036 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061458684504032135 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6687602996826172 *jet/accuracy_1_of_3‚0.9201767444610596 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16076616942882538 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9953755140304565 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13333332538604736 'particle/f_score‚0.6275896430015564 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16144168376922607 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934314489364624 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02293209172785282 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39033228158950806 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6642305254936218 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7477380037307739 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46101006865501404 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16705797612667084 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3160930275917053 +jet/accuracy_3_of_4‚0.08434997498989105 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7068651914596558 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17946648597717285 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924502372741699 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1730141043663025 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8959687352180481 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 +particle/sensitivity‚0.6029363870620728 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17093311250209808 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.030099088326096535 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8375160694122314 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255/MÉÛ¸ç Ì{ 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25937753915786743 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4221319556236267 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6358199119567871 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29634761810302734 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6911255717277527 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023957621306180954 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6263033747673035 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061458684504032135 *jet/accuracy_2_of_4‚0.7724812626838684 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906672239303589 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06341979652643204 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @¤· Ü9&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6validation_average_jet_accuracy‚0.3599169850349426 _step‚164 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6622573733329773 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906860589981079 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6576939821243286 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17080800235271454 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8572064638137817 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7473545074462891 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6070747971534729 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18291275203227997 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_4_of_4‚0.9172014594078064 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9881786108016968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06393878161907196 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 (particle/accuracy‚0.8463349342346191 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7534154653549194 particle/accuracy_2_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6830309629440308 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5890286564826965 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17615433037281036 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061458684504032135 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6687602996826172 *jet/accuracy_1_of_3‚0.9201767444610596 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16076616942882538 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9953755140304565 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13333332538604736 'particle/f_score‚0.6275896430015564 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16144168376922607 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934314489364624 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02293209172785282 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39033228158950806 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6642305254936218 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7477380037307739 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46101006865501404 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16705797612667084 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3160930275917053 +jet/accuracy_3_of_4‚0.08434997498989105 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7068651914596558 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17946648597717285 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924502372741699 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1730141043663025 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8959687352180481 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 +particle/sensitivity‚0.6029363870620728 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17093311250209808 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.030099088326096535 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8375160694122314 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965478777885437 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3921947479248047 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5594077706336975 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_2_of_3‚0.5259974002838135 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6687602996826172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6253212690353394 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7157258987426758 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 .particle/accuracy_3_of_4‚0.997619092464447 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 trainer/global_step‚1999 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.881232738494873 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5420169830322266 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15798570215702057 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1391916573047638 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17080800235271454 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.129060760140419 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867221117019653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11018378287553787 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.989795982837677 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7800543904304504 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0439470000565052 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7697246074676514 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6823735237121582 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7710772156715393 (jet/accuracy_3_of_3‚0.06414745002985 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791865348816 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 jet/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06292404979467392 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6991283297538757 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3893691897392273 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0581771545112133 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6915150880813599 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868274927139282 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6864567995071411 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0638013556599617 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7200415730476379 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10255154967308044 *loss/val_total_loss‚10.182209173871698 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1197049468755722 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886714220046997 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.51583331823349 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.530046820640564 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5248119831085205 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936407208442688 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09355971962213516 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843230753975291e+09 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2383744865655899 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18188419938087463 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.268325537443161 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7473544478416443 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03438909724354744 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_3_of_3‚0.9994933605194092 jet/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922657012939453 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7462018132209778 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17667332291603088 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5902751684188843 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7561303377151489 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.592757523059845 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7976943254470825 6"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7031210064888 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4034399688243866 +particle/specificity‚0.9129061102867126 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43714410066604614 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7267287969589233 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6473700404167175 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6409134864807129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.66053706407547 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18370914459228516 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948128461837769 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7030205726623535 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7423469424247742 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16531474888324738 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9943934082984924 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45627063512802124 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19310110807418823 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7875979542732239 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚4 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7448541522026062 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7976943254470825 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚3328.772805829 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7137147188186646 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7794456481933594 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955765604972839 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1500745564699173 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9695531129837036 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6304343342781067 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14822617173194885 *jet/accuracy_2_of_2‚0.1384061723947525 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14839135110378265 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161¤1Ëný“è ‚(:ˆ „¸êÒçæ¬ gpu.0.gpu‚28.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hk¹/é ‚(:¤ „¸êÒÖ…® network.sent‚ 3.668128e+06 disk.sda.out‚ 207.92578125# disk.dm-0.out‚44515.52001953125 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875$ memory_percent‚91.48697853088379 disk.sda2.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚44515.52001953125# proc.memory.rssMB‚ 3433.91796875) disk./.usagePercent‚39.08994272763289 disk.rbd2.out‚20.67041015625 disk.dm-0.in‚ 6753.921875& disk.nvme0n1.out‚44515.52001953125 disk.rbd2.in‚0.015625 network.recv‚848251 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 207.92578125( proc.memory.availableMB‚ 1394.7734375$ disk./.usageGB‚582.4683952331543 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.rbd1.in‚0.046875 disk.nvme0n1.in‚ 6753.921875* proc.memory.percent‚20.958971977233887# disk.rbd1.out‚17743.83642578125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 207.92578125! disk.nvme0c0n1.in‚ 6753.921875 cpu‚25.503429205375756 proc.cpu.threads‚17‰.Ý•”ê ‚(:‰ “¸êÒþ…¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O¬9&ë ‚(:› “¸êÒÍÍÝ® disk.drbd2165.out‚0 disk.sda2.in‚0# disk.dm-0.out‚46459.10498046875 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875$ memory_percent‚82.19153881072998 cpu‚25.499100264005037 network.sent‚ 3.725172e+06* disk./.usagePercent‚39.089857733131524# disk.rbd1.out‚17743.91455078125 disk.drbd2165.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 208.34375" proc.memory.rssMB‚ 2884.2734375 disk.rbd0.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚46459.10498046875 disk.sda2.out‚0) proc.memory.availableMB‚ 2917.73828125 disk.rbd1.in‚0.046875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 6753.921875 disk.nvme0n1.in‚ 6753.921875 disk.sda1.in‚0 disk.dm-1.out‚ 208.34375 proc.cpu.threads‚17 network.recv‚852556 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 6753.921875$ disk./.usageGB‚582.4671287536621 disk.rbd2.out‚21.04248046875 disk.sda3.out‚ 208.34375& disk.nvme0n1.out‚46459.10498046875) proc.memory.percent‚17.60420799255371û ÷)ì ‚(:ž ¢¸êÒùíÅ® disk.sda2.in‚0# disk.rbd1.out‚17743.91455078125 disk.rbd0.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚46476.83447265625 disk.sda.out‚ 210.6953125 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 6876.609375 disk.nvme0n1.in‚ 6876.609375 disk.dm-0.in‚ 6876.609375 network.sent‚ 3.730223e+06) disk./.usagePercent‚39.08944300092605 disk.rbd1.in‚0.046875 disk.dm-1.in‚ 0.01171875# memory_percent‚82.1918249130249) proc.memory.percent‚17.60420799255371 network.recv‚855804 disk.sda3.out‚ 210.6953125 disk.rbd2.out‚21.89794921875$ disk./.usageGB‚582.4609489440918 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0# disk.dm-0.out‚46476.83447265625 disk.sda1.in‚0" proc.memory.rssMB‚ 2884.2734375 cpu‚25.49866409129744 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚46476.83447265625 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 210.6953125) proc.memory.availableMB‚ 2917.69140625œ劔í ‚(:‰ ¢¸êÒ­À³® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚250.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìWÈú”î ‚(:‰ ±¸êÒœÀÞ­ gpu.0.gpu‚56.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚110.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$p…•-ï ‚(:¢ ±¸êÒš‡ÿ®) proc.memory.percent‚17.60420799255371 network.sent‚ 3.735306e+06 disk.dm-1.out‚ 211.1640625 disk.sda3.out‚ 211.1640625 proc.cpu.threads‚17$ disk./.usageGB‚582.4609069824219" disk.dm-0.out‚46485.0009765625 disk.sda.out‚ 211.1640625 disk.sda3.in‚ 0.01171875 network.recv‚859064 disk.sda2.out‚0# disk.rbd1.out‚17743.97705078125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 6935.41015625 disk.rbd2.in‚0.015625# disk.nvme0c0n1.in‚ 6935.41015625% disk.nvme0n1.out‚46485.0009765625 disk.sda.in‚ 0.01171875 cpu‚25.498221644386742 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 6935.41015625) disk./.usagePercent‚39.08944018484317 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2884.2734375 disk.rbd1.in‚0.046875 disk.rbd2.out‚22.10400390625' disk.nvme0c0n1.out‚46485.0009765625# memory_percent‚82.1918249130249) proc.memory.availableMB‚ 2917.69140625üì÷Hžð ‚(:“ À¸êÒÅæ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚297.732' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.54640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùø¡ßñ ‚(: À¸êÒÏÖ® disk.dm-1.in‚ 0.01171875 disk.dm-0.in‚ 6935.41015625 disk.dm-1.out‚ 211.46875! disk.nvme0n1.in‚ 6935.41015625 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.out‚ 46493.859375 disk.sda1.in‚0( proc.memory.availableMB‚ 2917.6796875" proc.memory.rssMB‚ 2884.2734375# disk.nvme0c0n1.out‚ 46493.859375 disk.rbd1.in‚0.046875 disk.rbd0.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 3.740351e+06$ disk./.usageGB‚582.4609298706055# disk.nvme0c0n1.in‚ 6935.41015625 disk.rbd0.in‚0) disk./.usagePercent‚39.08944172088838 disk.sda.out‚ 211.46875 disk.sda2.in‚0# disk.rbd1.out‚17743.97705078125) proc.memory.percent‚17.60420799255371 cpu‚25.497657210963407 disk.sda1.out‚0 disk.drbd2165.out‚0 network.recv‚862346 disk.sda3.out‚ 211.46875$ memory_percent‚82.19189643859863 disk.sda.in‚ 0.01171875! disk.nvme0n1.out‚ 46493.859375 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚22.10400390625ɽµ~’ò ‚(:‡ ϸêÒðñé¬ gpu.0.gpu‚24.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚338.845' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.769 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú? .ó ‚(:£ ϸêҀĽ®) disk./.usagePercent‚39.08943250461715" disk.dm-0.out‚46501.6533203125" disk.nvme0c0n1.in‚ 6936.2421875 network.sent‚ 3.745399e+06 disk.rbd2.out‚22.17138671875 disk.dm-1.out‚ 211.65234375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.497224685525712) proc.memory.availableMB‚ 2917.41796875" proc.memory.rssMB‚ 2884.2734375 disk.dm-0.in‚ 6936.2421875 disk.sda1.out‚0 disk.sda3.out‚ 211.65234375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 211.65234375 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 6936.2421875$ memory_percent‚82.19349384307861 proc.cpu.threads‚17 network.recv‚865672$ disk./.usageGB‚582.4607925415039 disk.sda2.out‚0) proc.memory.percent‚17.60420799255371 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚46501.6533203125% disk.nvme0n1.out‚46501.6533203125# disk.rbd1.out‚17744.11767578125 disk.rbd0.out‚0ÿ†Ú¦2ô - _wandbruntime‚3410  _runtime‚3410_k`Çõ Á & lr-AdamW/pg2‚0.0002693076371778348 trainer/global_step‚2049 % _timestamp‚1.7843231575599208e+09 _runtime‚3410.934541571 _step‚165 & lr-AdamW/pg1‚0.0002693076371778348†¥ÞÀÌö Æ _runtime‚3410.934541571 _step‚165 & lr-AdamW/pg1‚0.0002693076371778348 & lr-AdamW/pg2‚0.0002693076371778348 trainer/global_step‚2049 % _timestamp‚1.7843231575599208e+09¥‹ñCì2÷ - _wandbruntime‚3410  _runtime‚3410l;“’þ!ø èO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17486695 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159279816429568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17757010 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159279448338432 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161766494772224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17486701 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928806731264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚590799210430218 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588873703755776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1924162074120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5261944 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17757010 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927461341696 _runtime‚ 3410.93941975 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588873678327808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591056341314048 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17747037 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591057712877056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5825847 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17486701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648óH“7ð E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚590797840401928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1925506674442 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚591057712877056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589128906145792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5825617 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5261939 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591056341314048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843231575610125e+09BÅ·ó'ù íO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17757010 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159279448338432 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161766494772224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17486701 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928806731264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚590799210430218 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588873703755776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1924162074120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5261944 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17757010 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927461341696 _runtime‚ 3410.93941975 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588873678327808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591056341314048 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17747037 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591057712877056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5825847 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17486701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚590797840401928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1925506674442 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚591057712877056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589128906145792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5825617 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5261939 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591056341314048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843231575610125e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17486695 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159279816429568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¦©϶2ú - _wandbruntime‚3412  _runtime‚3412‰¾ù&û óM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1929731773952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11092857 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161844794034176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591078015674752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928386377216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591076645646350 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2487891720192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589406906875904 _step‚167 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1929731773952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5264482 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚2049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5264487 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589406880702464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11093090 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17494901 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159357302837248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚3412.333294835 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589406880702464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17755515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159356934746112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 % _timestamp‚1.7843231589583285e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5828603 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1925085540366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17494907 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚589151560105984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35260409 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ƒþ&ü øM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17494907 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2487891720192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589406906875904 _step‚167 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1929731773952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5264482 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚2049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5264487 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589406880702464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11093090 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17494901 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159357302837248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚3412.333294835 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589406880702464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17755515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159356934746112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 % _timestamp‚1.7843231589583285e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5828603 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1925085540366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17494907 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚589151560105984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35260409 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1929731773952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11092857 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161844794034176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591078015674752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928386377216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591076645646350 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17896995 _step‚170 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18182762 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975122661888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17897001 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1973777272320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5965120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605002693750272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚605002693750272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603028942651392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18182762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11354214 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163153768722432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17897001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18172789 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602767792664576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚2099 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚3490.179532049 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975122661888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17896995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605004065313280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5389089 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18172789 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2547011403776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2546976867328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604739536452126 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚605004065313280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚603028916477952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603028916477952 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5389094 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚603028916477952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603028916477952 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5389094 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17896995 _step‚170 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18182762 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975122661888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17897001 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1973777272320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5965120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605002693750272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚605002693750272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603028942651392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18182762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11354214 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163153768722432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17897001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627ªFû\2 - _wandbruntime‚3491  _runtime‚3491l²7 ‘ òM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1974702307840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11359203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18191254 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚171 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165779431652352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605016971668258 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_trauOñÞin_batch_end/allocation.small_pool.freed‚18181267 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603045674442752 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚603306917208064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚3491.564485749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165779834272768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605018341696660 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603045649014784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603306943381504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17905207 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2548211051520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972667253908 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1976047704576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1974702307840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17905201 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚605282991086080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5391637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163231255130112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17905201 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18181267 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11359456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18191254 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5391632 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640C&áý&’ ÷M F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165779431652352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605016971668258 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18181267 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603045674442752 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚603306917208064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚3491.564485749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165779834272768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605018341696660 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603045649014784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603306943381504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17905207 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2548211051520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972667253908 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1976047704576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1974702307840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17905201 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚605282991086080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5391637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163231255130112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17905201 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18181267 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11359456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18191254 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5391632 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1974702307840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11359203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18191254 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚171 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0«2ŒŒ2“ - _wandbruntime‚3491  _runtime‚3491TÞú ” ÿ /loss/FBt1/detection_loss‚0.5676697230711892 &loss/total_loss‚10.422167838165723 trainer/global_step‚2099 1loss/FBt2/assignment_loss‚0.07477349934970755 _runtime‚3491.650820361 /loss/FRt2/assignment_loss‚2.276969125284951 1loss/FBt1/assignment_loss‚0.04929519695866831 .loss/FRt2/detection_loss‚0.644797570061093 1loss/SRqqt1/detection_loss‚0.6984255810721715 % _timestamp‚1.7843232382764876e+09 0loss/FRt1/assignment_loss‚1.0368488087422318 /loss/FBt2/detection_loss‚0.9989342904063286 epoch‚5 _step‚172 2loss/SRqqt1/assignment_loss‚0.6184322633005204 2loss/SRqqt2/assignment_loss‚0.7661034116294345 .loss/FRt1/detection_loss‚1.528215201744051 1loss/SRqqt2/detection_loss‚1.1617031665453752ÖwE • „ 1loss/FBt1/assignment_loss‚0.04929519695866831 .loss/FRt2/detection_loss‚0.644797570061093 1loss/SRqqt1/detection_loss‚0.6984255810721715 % _timestamp‚1.7843232382764876e+09 0loss/FRt1/assignment_loss‚1.0368488087422318 /loss/FBt2/detection_loss‚0.9989342904063286 epoch‚5 _step‚172 2loss/SRqqt1/assignment_loss‚0.6184322633005204 2loss/SRqqt2/assignment_loss‚0.7661034116294345 .loss/FRt1/detection_loss‚1.528215201744051 1loss/SRqqt2/detection_loss‚1.1617031665453752 /loss/FBt1/detection_loss‚0.5676697230711892 &loss/total_loss‚10.422167838165723 trainer/global_step‚2099 1loss/FBt2/assignment_loss‚0.07477349934970755 _runtime‚3491.650820361 /loss/FRt2/assignment_loss‚2.276969125284951¬J˜ØEž– ‚(:“ ©¹êÒÁ¬¨® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚292.063' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.41259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íI3— ‚(:¨ ©¹êÒ”€û¯ disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚22.95068359375 disk.drbd2165.out‚0& disk.nvme0n1.out‚46546.39208984375# memory_percent‚82.1953296661377 cpu‚25.494361193704332 disk.rbd0.in‚0 disk.sda3.out‚ 213.27734375 disk.rbd1.in‚0.046875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda.out‚ 213.27734375# disk.rbd1.out‚17744.32080078125* disk./.usagePercent‚39.089484474146595 disk.dm-1.in‚ 0.01171875( disk.nvme0c0n1.out‚46546.39208984375 disk.sda.in‚ 0.01171875" proc.memory.rssMB‚ 2884.2734375 network.recv‚887312 disk.sda1.out‚0$ disk./.usageGB‚582.4615669250488 disk.dm-1.out‚ 213.27734375! disk.nvme0n1.in‚ 7211.55859375) proc.memory.percent‚17.60420799255371 network.sent‚ 3.834008e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7211.55859375# disk.dm-0.out‚46546.39208984375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7211.55859375( proc.memory.availableMB‚ 2917.1171875ó·uMž˜ ‚(:“ ¸¹êÒ˜Ù¼­ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚269.93' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.986000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’€ "™ ‚(:— ¸¹êÒ¤›Ò® network.recv‚891554 disk.rbd0.in‚0 disk.dm-0.in‚ 7211.5625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875) proc.memory.percent‚17.60420799255371$ disk.nvme0n1.out‚46553.146484375 disk.sda1.out‚0* disk./.usagePercent‚39.089490618327424 disk.rbd0.out‚0 disk.rbd2.out‚22.95068359375) proc.memory.availableMB‚ 2916.98046875& disk.nvme0c0n1.out‚46553.146484375 disk.sda2.out‚0$ memory_percent‚82.19616413116455" proc.memory.rssMB‚ 2884.2734375 disk.rbd1.in‚0.046875 disk.dm-1.in‚ 0.01171875$ disk./.usageGB‚582.4616584777832# disk.rbd1.out‚17744.32861328125 disk.rbd2.in‚0.015625! disk.dm-0.out‚46553.146484375 disk.sda3.out‚ 213.45703125 network.sent‚ 3.897189e+06 disk.nvme0c0n1.in‚ 7211.5625 disk.nvme0n1.in‚ 7211.5625 disk.sda.out‚ 213.45703125 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 213.45703125 cpu‚25.49383912453235 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 proc.cpu.threads‚17륾ᔚ ‚(:‰ ǹêÒ®Ú¬ gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚48.0 gpu.0.powerWatts‚336.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dïÌì&› ‚(:› ǹêÒ®¢‡¯ disk.sda2.in‚0 disk.dm-0.in‚ 7212.51953125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 3.902253e+06$ disk./.usageGB‚582.4615516662598 disk.sda.out‚ 213.78125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚46562.2177734375$ memory_percent‚82.19645023345947) proc.memory.availableMB‚ 2916.93359375 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 213.78125% disk.nvme0n1.out‚46562.2177734375 proc.cpu.threads‚17# disk.rbd1.out‚17744.40673828125 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2884.2734375) proc.memory.percent‚17.60420799255371 disk.sda.in‚ 0.01171875 disk.rbd2.out‚ 22.994140625 cpu‚25.493436317127788 network.recv‚894752 disk.sda2.out‚0 disk.rbd1.in‚0.046875" disk.dm-0.out‚46562.2177734375 disk.dm-1.out‚ 213.78125# disk.nvme0c0n1.in‚ 7212.51953125) disk./.usagePercent‚39.08948345011646 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 7212.51953125=Õ†”œ ‚(:‰ Ö¹êÒð›ã¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚283.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0McÀÈ ‚(:’ Ö¹êÒµœÎ®$ disk./.usageGB‚582.4613838195801 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875& disk.nvme0c0n1.out‚46574.029296875% proc.memory.availableMB‚ 2917.4375* disk./.usagePercent‚39.089472185784956 disk.rbd2.out‚ 22.994140625 disk.dm-0.in‚ 7213.3515625 disk.sda1.out‚0$ disk.nvme0n1.out‚46574.029296875 disk.sda2.out‚0$ memory_percent‚82.19337463378906 network.sent‚ 3.907343e+06 network.recv‚897994 disk.dm-1.out‚ 214.03125" disk.nvme0c0n1.in‚ 7213.3515625 disk.rbd1.in‚0.046875 proc.cpu.threads‚17 disk.sda3.out‚ 214.03125 cpu‚25.49290968759671! disk.dm-0.out‚46574.029296875 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.sda.in‚ 0.01171875# disk.rbd1.out‚17744.40673828125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 7213.3515625 disk.sda.out‚ 214.03125 disk.sda2.in‚0" proc.memory.rssMB‚ 2883.8515625* proc.memory.percent‚17.601633071899414 disk.sda1.in‚0‰“‚’ž ‚(:‡ å¹êÒ¶—í­ gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚340.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.013 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ªpC.Ÿ ‚(:£ å¹êÒÄÓ«® disk.rbd0.out‚0 cpu‚25.492524442547463$ disk.nvme0n1.out‚46578.580078125 disk.rbd1.in‚0.046875 disk.drbd2165.in‚0) proc.memory.percent‚17.60420799255371 disk.sda.out‚ 214.47265625! disk.dm-0.out‚46578.580078125 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 3.912384e+06$ disk./.usageGB‚582.4625396728516! disk.nvme0n1.in‚ 7319.55859375 disk.dm-0.in‚ 7319.55859375 disk.dm-1.out‚ 214.47265625) disk./.usagePercent‚39.08954975606783 disk.sda3.in‚ 0.01171875# disk.rbd1.out‚17744.56298828125 disk.rbd2.out‚23.05712890625 network.recv‚901402& disk.nvme0c0n1.out‚46578.580078125 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2884.2734375 disk.sda3.out‚ 214.47265625# disk.nvme0c0n1.in‚ 7319.55859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚82.19397068023682) proc.memory.availableMB‚ 2917.33984375ÆQ£r2  - _wandbruntime‚3568  _runtime‚3568—÷È¡  trainer/global_step‚2149 $ _timestamp‚1.784323315279055e+09 _runtime‚3568.653677407 _step‚173 ' lr-AdamW/pg1‚0.00026871686385267354 ' lr-AdamW/pg2‚0.00026871686385267354–8´Í¢ Ç _runtime‚3568.653677407 _step‚173 ' lr-AdamW/pg1‚0.00026871686385267354 ' lr-AdamW/pg2‚0.00026871686385267354 trainer/global_step‚2149 $ _timestamp‚1.784323315279055e+09­²&:82£ - _wandbruntime‚3568  _runtime‚3568@gj£¤ éO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11619140 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2606798626816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616661856145408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2021438592512 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚2149 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2606764090368 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618949046186496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2020093202944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6102896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2020093202944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169635255824384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167028457197568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618949046186496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚616928979156992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5516239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6102640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2021438592512 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169634853196800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843233152801569e+09 _runtime‚3568.658463152 ZKDeviceStatsMonitor.on_train_bat"ûL ch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18307301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2016636300336 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚616928952983552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5516244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616661881573376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18307301 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18307295 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616928952983552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18307295 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618950417749504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18608514 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618949046186496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2020093202944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6102896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2020093202944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169635255824384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167028457197568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618949046186496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚616928979156992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5516239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6102640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2021438592512 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169634853196800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843233152801569e+09 _runtime‚3568.658463152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18307301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2016636300336 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚616928952983552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5516244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616661881573376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18307301 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18307295 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616928952983552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18307295 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618950417749504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18608514 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18315507 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚3570.040693164 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36932513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18315501 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022363635200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2018904367016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18617006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚618957297690166 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617206953713664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11624434 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11624172 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169713536243712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2607995258880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2021018238464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5518782 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175çÈÝZ˜ª øM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚3570.040693164 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36932513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18315501 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022363635200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2018904367016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18617006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚618957297690166 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617206953713664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11624434 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11624172 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169713536243712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2607995258880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2021018238464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5518782 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18315507 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416¯ó§Œ2« - _wandbruntime‚3570  _runtime‚3570߬ ü 0loss/FBt2/assignment_loss‚0.0772746662615831 1loss/SRqqt2/detection_loss‚1.0151283979064687 /loss/FBt1/detection_loss‚0.7073484176480418 /loss/FBt2/detection_loss‚0.8158404064123277 $loss/total_loss‚9.90721938957213 _step‚176 /loss/FRt2/detection_loss‚0.6299023700608232 1loss/SRqqt1/detection_loss‚0.7370703482076688 epoch‚5 0loss/FRt1/assignment_loss‚0.8713187841432435 2loss/SRqqt2/assignment_loss‚0.7964105344217142 trainer/global_step‚2149 /loss/FRt2/assignment_loss‚2.156693131506443 2loss/SRqqt1/assignment_loss‚0.5667598051012689 .loss/FRt1/detection_loss‚1.482403136704566 % _timestamp‚1.7843233167503192e+09 _runtime‚ 3570.12467869 1loss/FBt1/assignment_loss‚0.05106939119798141û`öP­  1loss/FBt1/assignment_loss‚0.05106939119798141 0loss/FBt2/assignment_loss‚0.0772746662615831 1loss/SRqqt2/detection_loss‚1.0151283979064687 /loss/FBt1/detection_loss‚0.7073484176480418 /loss/FBt2/detection_loss‚0.8158404064123277 $loss/total_loss‚9.90721938957213 _step‚176 /loss/FRt2/detection_loss‚0.6299023700608232 1loss/SRqqt1/detection_loss‚0.7370703482076688 epoch‚5 0loss/FRt1/assignment_loss‚0.8713187841432435 2loss/SRqqt2/assignment_loss‚0.7964105344217142 trainer/global_step‚2149 /loss/FRt2/assignment_loss‚2.156693131506443 2loss/SRqqt1/assignment_loss‚0.5667598051012689 .loss/FRt1/detection_loss‚1.482403136704566 % _timestamp‚1.7843233167503192e+09 _runtime‚ 3570.12467869°T\)“® ‚(:ˆ ƒºêÒâÑ‘¬ gpu.0.gpu‚21.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚335.367' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚^*-¯ ‚(:¢ ƒºêÒé·à­ disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 215.16796875 disk.rbd1.in‚0.046875 disk.drbd2165.out‚0 disk.dm-1.out‚ 215.16796875) disk./.usagePercent‚39.08956281245208% disk.nvme0n1.out‚46598.1865234375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2884.27734375" disk.dm-0.out‚46598.1865234375 disk.drbd2165.in‚0 network.sent‚ 3.973646e+06 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 7321.34765625 disk.sda2.out‚0 disk.dm-0.in‚ 7321.34765625) proc.memory.percent‚17.60423183441162 network.recv‚916334 disk.sda2.in‚0 disk.rbd2.out‚ 23.1201171875 disk.sda1.out‚0& proc.memory.availableMB‚ 1944.71875 cpu‚25.49162576812637 proc.cpu.threads‚17$ disk./.usageGB‚582.4627342224121' disk.nvme0c0n1.out‚46598.1865234375 disk.sda1.in‚0$ memory_percent‚88.13037872314453! disk.nvme0n1.in‚ 7321.34765625 disk.sda.in‚ 0.01171875 disk.sda.out‚ 215.16796875# disk.rbd1.out‚17744.64111328125 disk.rbd0.out‚0ºÉ5° ‚(:ª ’ºêÒ·¨Ë® disk.dm-0.in‚ 7321.34765625# disk.dm-0.out‚46603.22900390625 disk.rbd1.in‚0.046875) proc.memory.availableMB‚ 1945.26953125# proc.memory.rssMB‚ 2883.87109375 network.recv‚920311# disk.nvme0c0n1.in‚ 7321.34765625 proc.cpu.threads‚17* disk./.usagePercent‚39.089527483412354& disk.nvme0n1.out‚46603.22900390625 network.sent‚ 4.026289e+06 disk.sda3.out‚ 215.38671875( disk.nvme0c0n1.out‚46603.22900390625 disk.sda.out‚ 215.38671875 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 23.1201171875$ disk./.usageGB‚582.4622077941895 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875# disk.rbd1.out‚17744.64111328125 disk.rbd0.out‚0 disk.dm-1.out‚ 215.38671875 disk.sda1.in‚0# memory_percent‚88.1270170211792* proc.memory.percent‚17.601752281188965 cpu‚25.491027374718296 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 7321.34765625 disk.rbd0.in‚0ì&É«Ÿ± ‚(:” ’ºêÒà”à® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚282.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.531200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°]5Q•² ‚(:Š ¡ºêÒ¿ÙÇ­ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚338.604' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v>˜Ï*³ ‚(:Ÿ ¡ºêÒ³àï¯ disk.dm-0.in‚ 7414.45703125 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 215.73828125# disk.nvme0c0n1.in‚ 7414.45703125# disk.nvme0n1.out‚46610.12109375 disk.rbd2.in‚0.015625 cpu‚25.490702090811318 network.recv‚923447 disk.dm-1.out‚ 215.73828125# proc.memory.rssMB‚ 2884.27734375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 7414.45703125# memory_percent‚88.1300687789917 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875) proc.memory.availableMB‚ 1944.76953125 network.sent‚ 4.030909e+06) disk./.usagePercent‚39.08954258785687) proc.memory.percent‚17.60423183441162# disk.rbd1.out‚17744.70361328125 disk.dm-0.out‚46610.12109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚582.4624328613281 disk.sda1.in‚0% disk.nvme0c0n1.out‚46610.12109375 disk.sda.out‚ 215.73828125 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.rbd2.out‚ 23.1201171875ï·Cò”´ ‚(:‰ °ºêÒ๛¯ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Æ®b/µ ‚(:¤ °ºêÒžáÍ° disk.dm-0.in‚ 7415.2890625" disk.dm-0.out‚46618.8583984375 disk.sda1.in‚0% disk.nvme0n1.out‚46618.8583984375 disk.sda2.in‚0# disk.rbd1.out‚17744.70361328125 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2884.27734375 network.sent‚ 4.039494e+06$ disk./.usageGB‚582.4624176025391 disk.dm-1.out‚ 215.97265625 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 proc.cpu.threads‚17$ memory_percent‚88.12639713287354 disk.nvme0n1.in‚ 7415.2890625 disk.sda.out‚ 215.97265625' disk.nvme0c0n1.out‚46618.8583984375 disk.sda.in‚ 0.01171875 disk.sda2.out‚0) proc.memory.percent‚17.60423183441162 disk.dm-1.in‚ 0.01171875" disk.nvme0c0n1.in‚ 7415.2890625 disk.rbd2.out‚ 23.1201171875) proc.memory.availableMB‚ 1945.37109375 cpu‚25.490182257970382 network.recv‚933921* disk./.usagePercent‚39.089541563826735 disk.sda3.out‚ 215.97265625 disk.rbd1.in‚0.046875 disk.rbd0.in‚0­b¦”¶ ‚(:‰ ¿ºêÒ‘ñ¬ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚296.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ncg1· ‚(:¦ ¿ºêÒϾú­ disk.sda2.out‚0 disk.rbd0.out‚0 network.sent‚ 4.044544e+06 disk.nvme0n1.in‚ 7415.2890625 disk.rbd2.out‚ 23.1201171875# disk.dm-0.out‚46621.41748046875 network.recv‚937675) disk./.usagePercent‚39.08954514793221" disk.nvme0c0n1.in‚ 7415.2890625 disk.sda.out‚ 216.23828125$ memory_percent‚88.12613487243652# proc.memory.rssMB‚ 2883.85546875 cpu‚25.489852115170702$ disk./.usageGB‚582.4624710083008 disk.sda3.out‚ 216.23828125 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 7415.2890625 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda2.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 216.23828125# disk.rbd1.out‚17744.78173828125 disk.rbd0.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1945.4140625* proc.memory.percent‚17.601656913757324( disk.nvme0c0n1.out‚46621.41748046875& disk.nvme0n1.out‚46621.41748046875óýJ2¸ - _wandbruntime‚3647  _runtime‚3647Tæ3åǹ Á & lr-AdamW/pg1‚0.0002684087808748872 & lr-AdamW/pg2‚0.0002684087808748872 trainer/global_step‚2199 % _timestamp‚1.7843233940138674e+09 _runtime‚3647.389444762 _step‚177š‘x̺ Æ & lr-AdamW/pg1‚0.0002684087808748872 & lr-AdamW/pg2‚0.0002684087808748872 trainer/global_step‚2199 % _timestamp‚1.7843233940138674e+09 _runtime‚3647.389444762 _step‚177±‡ŠŽ2» - _wandbruntime‚3647  _runtime‚3647VÂQî'¼ èO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062873413444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚2199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18717595 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19034266 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630829015662592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067754523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19024293 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632895398622720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19034266 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚178 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19024293 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630828989489152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632895398622720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6240156 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067754523136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170902409490432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6240405 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630555945054208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2066409133568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5643389 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚632620188495942 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632618818467652 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630828989489152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666650266624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18717595 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666617827328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0õÖ©ó'½ íO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173569427848192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170902777581568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632896770185728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3647.389557495 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚632896770185728 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5643394 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630829015662592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173569027317760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11883545 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18717601 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2066409133568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666617827328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062873413444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚2199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18717595 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19034266 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630829015662592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067754523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19024293 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632895398622720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19034266 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚178 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19024293 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630828989489152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632895398622720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6240156 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067754523136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170902409490432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6240405 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630555945054208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2066409133568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5643389 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚632620188495942 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632618818467652 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630828989489152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666650266624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18717595 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632898993740476 trainer/global_step‚2199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 % _timestamp‚1.7843233953984673e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630833826832384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173647713150976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667849684992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063796879690 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170979895898112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19032771 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630833852260352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633175695958528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2065141480124 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633174324388352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2067334169088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11888844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173648113674240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18725801 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11888844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173648113674240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18725801 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632898993740476 trainer/global_step‚2199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 % _timestamp‚1.7843233953984673e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630833826832384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173647713150976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667849684992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063796879690 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170979895898112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19032771 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630833852260352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633175695958528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2065141480124 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633174324388352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2067334169088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644729025994752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚644729052168192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚644450059390976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19450045 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12148911 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460018 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644729025994752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177503893945344 =7DeviceStatsMonitor.on?¹j_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2109110526552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19127895 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646841751058944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6378134 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646843122621952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚646841751058944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110455126874 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38587919 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843234726198573e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174777097965568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646560514517850 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19450045Š—Mœô'Ó îO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6378134 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646843122621952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚646841751058944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110455126874 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38587919 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843234726198573e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174777097965568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646560514517850 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19450045 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644729052168192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460018 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6378367 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646559144489560 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174776729874432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2726795979776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19127901 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚646843122621952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2112725064192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19127895 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2112725064192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5770539 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚644450033963008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644729025994752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚644729052168192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚644450059390976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19450045 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12148911 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460018 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644729025994752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177503893945344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2109110526552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19127895 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646841751058944¶YOìè2Ô - _wandbruntime‚3727  _runtime‚3727„¢µlù&Õ óM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174854584373248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646837949733982 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19136101 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19136107 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5773087 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647122048394752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6380886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645007052898304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _runtime‚3727.380421896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19458523 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645007026724864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12153968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177582185457664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚2249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19458523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚644727941169152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177582585980928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2110033992798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843234740055177e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19136101 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647120676824576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645007026724864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645007052898304ÖŽÁ .Ö øM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19136107 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5773087 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647122048394752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6380886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.­•êyÐallocated‚645007052898304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _runtime‚3727.380421896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19458523 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645007026724864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12153968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177582185457664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚2249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19458523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚644727941169152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177582585980928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2110033992798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843234740055177e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19136101 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647120676824576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645007026724864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645007052898304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174854584373248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646837949733982 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19136101 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624·-8u¢2× - _wandbruntime‚3727  _runtime‚3727À- Ø € trainer/global_step‚2249 % _timestamp‚1.7843234740919514e+09 _runtime‚3727.466442479 _step‚184 0loss/FRt1/assignment_loss‚0.8743665363596773 2loss/SRqqt1/assignment_loss‚0.5719958298541635 /loss/FBt2/detection_loss‚1.0497879341244698 0loss/FBt2/assignment_loss‚0.5312579180459125 /loss/FRt2/detection_loss‚0.5492661488447478 &loss/total_loss‚10.140760003324512 2loss/SRqqt2/assignment_loss‚0.6027215290812536 1loss/SRqqt2/detection_loss‚1.0817434763811555 /loss/FBt1/detection_loss‚0.5725658127682781 epoch‚5 /loss/FRt2/assignment_loss‚2.131058695375408 1loss/FBt1/assignment_loss‚0.04930649645844051 /loss/FRt1/detection_loss‚1.4770886155102274 1loss/SRqqt1/detection_loss‚0.6496010105207781?¾ˆ+ Ù … 0loss/FRt1/assignment_loss‚0.8743665363596773 2loss/SRqqt1/assignment_loss‚0.5719958298541635 /loss/FBt2/detection_loss‚1.0497879341244698 0loss/FBt2/assignment_loss‚0.5312579180459125 /loss/FRt2/detection_loss‚0.5492661488447478 &loss/total_loss‚10.140760003324512 2loss/SRqqt2/assignment_loss‚0.6027215290812536 1loss/SRqqt2/detection_loss‚1.0817434763811555 /loss/FBt1/detection_loss‚0.5725658127682781 epoch‚5 /loss/FRt2/assignment_loss‚2.131058695375408 1loss/FBt1/assignment_loss‚0.04930649645844051 /loss/FRt1/detection_loss‚1.4770886155102274 1loss/SRqqt1/detection_loss‚0.6496010105207781 trainer/global_step‚2249 % _timestamp‚1.7843234740919514e+09 _runtime‚3727.466442479 _step‚184¸"0Ha’Ú ‚(:‡ ™»êÒÃÁ–® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚286.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó{…þ#Û ‚(:˜ ™»êÒÌ‘·¯ disk.rbd2.in‚0.015625! disk.nvme0n1.in‚ 7772.35546875( proc.memory.availableMB‚ 1945.0390625 cpu‚25.487414518962527 network.sent‚ 4.133646e+06 disk.rbd0.in‚0 disk.rbd2.out‚ 23.1201171875 disk.dm-0.in‚ 7772.35546875 disk.dm-1.out‚217.8125# memory_percent‚88.1284236907959 disk.sda.in‚ 0.01171875 network.recv‚960692$ disk./.usageGB‚582.4624862670898 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚217.8125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 7772.35546875% disk.nvme0n1.out‚46668.9423828125 disk.sda2.out‚0) disk./.usagePercent‚39.08954617196235" disk.dm-0.out‚46668.9423828125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0.046875# proc.memory.rssMB‚ 2884.27734375) proc.memory.percent‚17.60423183441162' disk.nvme0c0n1.out‚46668.9423828125# disk.rbd1.out‚17744.98486328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.out‚217.8125 disk.sda2.in‚0¸`‹ŸÜ ‚(:” ¨»êÒð줬 gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚331.588' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.31760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÇÀ¥Ý ‚(:’ ¨»êÒÕ³¢® proc.cpu.threads‚17 network.recv‚965595) disk./.usagePercent‚39.08956511651988# disk.nvme0n1.out‚46678.06640625 disk.rbd1.in‚0.046875 disk.rbd0.in‚0 disk.dm-0.out‚46678.06640625( proc.memory.availableMB‚ 1944.7421875 network.sent‚ 4.196142e+06 disk.rbd2.in‚0.015625 disk.sda.out‚218.0625$ memory_percent‚88.13023567199707# proc.memory.rssMB‚ 2884.27734375 cpu‚25.48708354312391 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚218.0625 disk.sda3.in‚ 0.01171875% disk.nvme0c0n1.out‚46678.06640625 disk.rbd0.out‚0 disk.dm-0.in‚ 8009.00390625 disk.sda1.out‚0! disk.nvme0n1.in‚ 8009.00390625# disk.rbd1.out‚17744.98486328125 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚218.0625# disk.nvme0c0n1.in‚ 8009.00390625 disk.rbd2.out‚ 23.1201171875 disk.drbd2165.in‚0) proc.memory.percent‚17.60423183441162$ disk./.usageGB‚582.4627685546875 disk.sda2.out‚0 disk.sda1.in‚0âÀ‘G”Þ ‚(:‰ ·»êÒÛ™û® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£†#ß ‚(:˜ ·»êÒ澯 disk.sda.out‚218.875 disk.sda2.in‚0 disk.drbd2165.in‚0 network.sent‚ 4.201225e+06) disk./.usagePercent‚39.08965062303632 disk.rbd2.in‚0.015625 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8009.00390625$ disk./.usageGB‚582.4640426635742 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0.046875# proc.memory.rssMB‚ 2884.27734375 disk.dm-0.in‚ 8009.00390625 disk.sda1.out‚0# disk.rbd1.out‚17745.06298828125' proc.memory.availableMB‚ 1944.796875 disk.rbd2.out‚ 23.1201171875# disk.dm-0.out‚46684.20068359375! disk.nvme0n1.in‚ 8009.00390625 disk.sda.in‚ 0.01171875 network.recv‚969092 disk.sda3.in‚ 0.01171875 cpu‚25.486529044019196 disk.rbd0.in‚0& disk.nvme0n1.out‚46684.20068359375 disk.sda2.out‚0) proc.memory.percent‚17.60423183441162 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚218.875$ memory_percent‚88.12990188598633 disk.sda3.out‚218.875( disk.nvme0c0n1.out‚46684.20068359375 proc.cpu.threads‚17G©3 à ‚(:• Æ»êÒÔüö« gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚185.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Xå)á ‚(:ž Æ»êÒÞ¹­ disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875# proc.memory.rssMB‚ 2884.27734375 network.sent‚ 4.206516e+06 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.out‚ 219.84375$ disk./.usageGB‚582.4639472961426# disk.rbd1.out‚17745.06298828125 disk.rbd2.out‚ 23.1201171875$ memory_percent‚88.13326358795166) proc.memory.availableMB‚ 1944.24609375 disk.rbd1.in‚0.046875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚46696.34228515625 disk.sda.in‚ 0.01171875 disk.sda.out‚ 219.84375 disk.dm-0.in‚ 8009.8359375" disk.nvme0c0n1.in‚ 8009.8359375 disk.nvme0n1.in‚ 8009.8359375& disk.nvme0n1.out‚46696.34228515625 cpu‚25.486211281989593 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875# disk.dm-0.out‚46696.34228515625* disk./.usagePercent‚39.089644222847966 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 219.84375 disk.rbd0.out‚0) proc.memory.percent‚17.60423183441162 network.recv‚972503°á‘í’â ‚(:‡ Õ»êÒÞ…­­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚121.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ǭr±2ã ‚(:§ Õ»êÒ½Õ…¯# proc.memory.rssMB‚ 2884.27734375 network.recv‚976152 disk.rbd2.out‚ 23.1201171875# disk.dm-0.out‚46704.85986328125 disk.dm-1.out‚ 220.5703125( disk.nvme0c0n1.out‚46704.85986328125 disk.sda2.in‚0 network.sent‚ 4.211558e+06* disk./.usagePercent‚39.089630398441116 disk.sda1.out‚0! disk.nvme0n1.in‚ 8010.79296875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0.046875 cpu‚25.485719576215608 proc.cpu.threads‚17# disk.rbd1.out‚17745.12548828125) proc.memory.availableMB‚ 1944.73828125 disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 8010.79296875 disk.sda.in‚ 0.01171875) proc.memory.percent‚17.60423183441162 disk.dm-0.in‚ 8010.79296875 disk.rbd0.in‚0$ memory_percent‚88.13025951385498 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 220.5703125$ disk./.usageGB‚582.4637413024902 disk.drbd2165.out‚0 disk.sda3.out‚ 220.5703125& disk.nvme0n1.out‚46704.85986328125 disk.sda2.out‚0m É­2ä - _wandbruntime‚3804  _runtime‚3804jvàìÇå Á % _timestamp‚1.7843235510222585e+09 _runtime‚3804.396852568 _step‚185 & lr-AdamW/pg1‚0.0002677673212256018 & lr-AdamW/pg2‚0.0002677673212256018 trainer/global_step‚2299I”(áÌæ Æ _runtime‚3804.396852568 _step‚185 & lr-AdamW/pg1‚0.0002677673212256018 & lr-AdamW/pg2‚0.0002677673212256018 trainer/global_step‚2299 % _timestamp‚1.7843235510222585e+09¹zPç2ç - _wandbruntime‚3804  _runtime‚3804Ö”Øï'è éO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843235510233727e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚658629062500352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚658629062500352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚3804.398131904 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660499470511468 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚2299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178651050258432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚658344122871808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5897694 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6516344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12414033 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚660789475058176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19875797 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660788103495168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19885770 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2155347639660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181438415413248 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19875797 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5897689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19538201 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660500840539758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚658629088673792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159040994816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6516576 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786997063680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181438014882816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚658344148299776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚660788103495168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786964624384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19885770 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19538201 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159040994816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178651418349568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2156692239982 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660789475058176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12414270 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19538195 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚186 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19538195 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚658629088673792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2160386384384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5897694 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6516344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12414033 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚660789475058176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19875797 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660788103495168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19885770 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2155347639660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181438415413248 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19875797 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5897689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19538201 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660500840539758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚658629088673792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159040994816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6516576 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786997063680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181438014882816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚658344148299776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚660788103495168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786964624384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19885770 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19538201 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159040994816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178651418349568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2156692239982 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660789475058176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12414270 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19538195 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_p ÙMä.ool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚186 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19538195 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚658629088673792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2160386384384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19894262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660779645784292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2788165426176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19894262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658907089403904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5900237 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161311427072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19884275 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚658907063230464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19546401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6519106 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚661068400830976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178728536666112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2159966030336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39430676 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19546407 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178728904757248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2788197858304 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419338 _runtime‚3805.783851786Äe¡¨þ&ì øM % _timestamp‚1.7843235524089403e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181516702092288 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19546401 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6519332 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658622004649984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12419569 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658907063230464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660778275755890 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚658907089403904 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19546401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6519106 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚661068400830976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178728536666112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2159966030336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39430676 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19546407 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178728904757248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2788197858304 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419338 _runtime‚3805.783851786 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19884275 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658622030077952 trainer/global_step‚2299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5900232 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2159966030336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19894262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660779645784292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2788165426176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19894262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658907089403904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5900237 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161311427072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19884275 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚658907063230464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674734455931392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20301549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2206702315008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12679002 WKDeviceStatsMonitor.on_train_batch.¡ Ñ!_start/inactive_split.small_pool.allocated‚6654158 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674441166561666 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843236296129727e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚672529099005952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672238237208576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19948501 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12678764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚674734455931392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2206702315008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19948495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182525370642432 _runtime‚3882.991392054 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19948501 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2847178803200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672238211780608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6024844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20311522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672529125179392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185372549445632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2202929353090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672529099005952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6653925 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674439796533376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚2349 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6024839 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20311522 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19948495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182525370642432 _runtime‚3882.991392054 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19948501 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2847178803200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672238211780608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6024844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20311522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672529125179392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185372549445632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2202929353090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672529099005952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6653925 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674439796533376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚2349 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6024839 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20311522 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674734455931392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20301549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2206702315008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12679002 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6654158 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674441166561666 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843236296129727e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚672529099005952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672238237208576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19948501 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12678764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚674734455931392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2206702315008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672807099736064 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚191 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20310027 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672516093558784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672516118986752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675014753267200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19956701 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19956701 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12684277 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2202508219014 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185451638005760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2206281960960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19956707 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6027382 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675013381697024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6656665 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2848412864512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2206281960960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6027387 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20310027 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12684047 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672807125909504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚674718601777798 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20310027 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12684047 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672807125909504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚674718601777798 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672807099736064 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚191 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20310027 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672516093558784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672516118986752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675014753267200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19956701 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19956701 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12684277 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2202508219014 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185451638005760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2206281960960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19956707 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6027382 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675013381697024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6656665 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545¿¸f[‚2ƒ - _wandbruntime‚3884  _runtime‚3884ˆ}.=„ þ % _timestamp‚1.7843236310833344e+09 _runtime‚3884.457725631 _step‚192 2loss/SRqqt1/assignment_loss‚0.5972358264668804 0loss/FBt1/assignment_loss‚0.0420703890013924 /loss/FRt2/detection_loss‚0.6725844154164685 /loss/FBt2/detection_loss‚0.9918469859716984 0loss/FRt1/assignment_loss‚1.0939935660537552 1loss/FBt2/assignment_loss‚0.12050095072167394 /loss/FRt1/detection_loss‚1.6400021740952142 /loss/FRt2/assignment_loss‚2.193421214702992 2loss/SRqqt2/assignment_loss‚0.6764209211247859 1loss/SRqqt1/detection_loss‚0.6339814790550412 1loss/SRqqt2/detection_loss‚1.1183571697232857 /loss/FBt1/detection_loss‚0.6175910929405112 trainer/global_step‚2349 $loss/total_loss‚10.3980061852737 epoch‚5Z¯± … ƒ _runtime‚3884.457725631 _step‚192 2loss/SRqqt1/assignment_loss‚0.5972358264668804 0loss/FBt1/assignment_loss‚0.0420703890013924 /loss/FRt2/detection_loss‚0.6725844154164685 /loss/FBt2/detection_loss‚0.9918469859716984 0loss/FRt1/assignment_loss‚1.0939935660537552 1loss/FBt2/assignment_loss‚0.12050095072167394 /loss/FRt1/detection_loss‚1.6400021740952142 /loss/FRt2/assignment_loss‚2.193421214702992 2loss/SRqqt2/assignment_loss‚0.6764209211247859 1loss/SRqqt1/detection_loss‚0.6339814790550412 1loss/SRqqt2/detection_loss‚1.1183571697232857 /loss/FBt1/detection_loss‚0.6175910929405112 trainer/global_step‚2349 $loss/total_loss‚10.3980061852737 epoch‚5 % _timestamp‚1.7843236310833344e+09ÀV+(“† ‚(:ˆ ¯¼êÒ…¨Œ¬ gpu.0.gpu‚17.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.047' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x·åz/‡ ‚(:¤ ¯¼êÒµ¼Ç®) disk./.usagePercent‚39.08962553429796# disk.rbd1.out‚17747.95361328125 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 222.43359375# memory_percent‚88.1319522857666( proc.memory.availableMB‚ 1944.4609375 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2884.27734375$ disk./.usageGB‚582.4636688232422 disk.rbd1.in‚0.046875 disk.rbd0.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 8257.93359375% disk.nvme0n1.out‚46751.2060546875 disk.sda.in‚ 0.01171875 disk.sda.out‚ 222.43359375 disk.rbd2.out‚ 23.1201171875 disk.sda2.in‚0 disk.dm-0.in‚ 8257.93359375 disk.sda3.in‚ 0.01171875) proc.memory.percent‚17.60423183441162 cpu‚25.483497244107138 proc.cpu.threads‚17 network.sent‚ 4.300128e+06 network.recv‚998864 disk.sda1.out‚0 disk.dm-1.out‚ 222.43359375' disk.nvme0c0n1.out‚46751.2060546875! disk.nvme0n1.in‚ 8257.93359375 disk.rbd2.in‚0.015625" disk.dm-0.out‚46751.2060546875 disk.rbd0.in‚0ñ¡’ˆ ‚(:‡ ¾¼êÒ¢¹Ö® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚263.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂM„ '‰ ‚(:œ ¾¼êÒɨ¼¯& disk.nvme0n1.out‚46755.68798828125 disk.sda.in‚ 0.01171875 disk.rbd1.in‚0.046875 disk.rbd2.in‚0.015625# disk.dm-0.out‚46755.68798828125( proc.memory.availableMB‚ 1944.8984375# proc.memory.rssMB‚ 2884.27734375 disk.sda3.in‚ 0.01171875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚46755.68798828125 disk.rbd0.out‚0 disk.dm-0.in‚ 8257.93359375) proc.memory.percent‚17.60423183441162 network.recv‚ 1.003681e+06# disk.nvme0c0n1.in‚ 8257.93359375! disk.nvme0n1.in‚ 8257.93359375 disk.sda.out‚223.25# disk.rbd1.out‚17747.96142578125 disk.rbd0.in‚0 cpu‚25.483038389458066 proc.cpu.threads‚17$ disk./.usageGB‚582.4626388549805 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚223.25 disk.rbd2.out‚ 23.1201171875) disk./.usagePercent‚39.08955641226372 disk.drbd2165.in‚0 disk.dm-1.out‚223.25$ memory_percent‚88.12928199768066 network.sent‚ 4.363719e+06 disk.sda2.in‚0 (†°”Š ‚(:‰ ͼêÒÜÛ¬ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚325.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÌ ‹ ‚(:“ ͼêÒ¦Éî® disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 8258.765625) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17 disk.nvme0n1.in‚ 8258.765625 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875) disk./.usagePercent‚39.08956383648221( disk.nvme0c0n1.out‚46763.36572265625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2884.27734375 network.sent‚ 4.368242e+06 disk.rbd0.in‚0 disk.rbd2.in‚0.015625$ memory_percent‚88.13233375549316 disk.sda.out‚223.5 network.recv‚ 1.007348e+06 disk.rbd2.out‚ 23.1201171875 disk.dm-1.out‚223.5( proc.memory.availableMB‚ 1944.3984375 cpu‚25.482793797095187$ disk./.usageGB‚582.4627494812012! disk.nvme0c0n1.in‚ 8258.765625& disk.nvme0n1.out‚46763.36572265625 disk.sda2.in‚0# disk.dm-0.out‚46763.36572265625 disk.sda3.out‚223.5 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd1.in‚0.046875# disk.rbd1.out‚17748.02392578125 disk.rbd0.out‚0è}פ1Œ ‚(:¦ ܼêÒÚÌ­ disk.dm-0.in‚ 8303.73046875 disk.sda1.out‚0$ memory_percent‚88.12623023986816( proc.memory.availableMB‚ 1945.3984375$ disk./.usageGB‚582.4621315002441 disk.dm-1.out‚ 223.8046875& disk.nvme0c0n1.out‚46769.103515625 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.sda1.in‚0) proc.memory.percent‚17.59812831878662 network.sent‚ 4.373329e+06 network.recv‚ 1.010944e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875$ disk.nvme0n1.out‚46769.103515625 disk.drbd2165.out‚0 cpu‚25.482307178248533* disk./.usagePercent‚39.089522363261665# disk.nvme0c0n1.in‚ 8303.73046875 disk.sda.out‚ 223.8046875 disk.sda2.out‚0 disk.rbd2.out‚ 23.1201171875! disk.dm-0.out‚46769.103515625# proc.memory.rssMB‚ 2883.27734375! disk.nvme0n1.in‚ 8303.73046875 disk.sda3.out‚ 223.8046875 disk.sda.in‚ 0.01171875# disk.rbd1.out‚17748.02392578125 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0T0˜x” ‚(:‰ ܼêÒþÞª® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0::ôΓŽ ‚(:ˆ ë¼êÒ»’á« gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚48.0 gpu.0.powerWatts‚332.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡žm# ‚(:˜ ë¼êÒºãί) proc.memory.percent‚17.60423183441162 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 8304.6875 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.nvme0c0n1.in‚ 8304.6875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1944.63671875) disk./.usagePercent‚39.08951954717879 disk.nvme0n1.in‚ 8304.6875# disk.rbd1.out‚17748.13330078125 disk.rbd0.in‚0 network.sent‚ 4.37745e+06$ disk./.usageGB‚582.4620895385742 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd2.out‚ 23.1201171875 disk.drbd2165.in‚0$ memory_percent‚88.13087940216064 proc.cpu.threads‚17% disk.nvme0c0n1.out‚46776.93359375 disk.sda2.out‚0 disk.rbd1.in‚0.046875 disk.dm-0.out‚46776.93359375 disk.dm-1.out‚ 224.08984375# proc.memory.rssMB‚ 2884.27734375 network.recv‚ 1.013697e+06 disk.sda.out‚ 224.08984375 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚25.48203212331019 disk.sda3.out‚ 224.08984375# disk.nvme0n1.out‚46776.933593752jyÖ” ‚(:‰ ú¼êÒž¥ª® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.003' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ €8‘ ‚(:­ ú¼êÒÛ÷ί$ disk./.usageGB‚582.4623641967773# disk.rbd1.out‚17748.13330078125 disk.rbd2.in‚0.015625! disk.nvme0n1.in‚ 8305.51953125 cpu‚25.481597902881386) disk./.usagePercent‚39.08953797972126# proc.memory.rssMB‚ 2884.27734375 disk.rbd0.in‚0# disk.dm-0.out‚46788.36767578125 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8305.51953125 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 224.44921875( disk.nvme0c0n1.out‚46788.36767578125 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 8305.51953125 disk.sda3.out‚ 224.44921875$ memory_percent‚88.13087940216064) proc.memory.availableMB‚ 1944.63671875 disk.rbd2.out‚ 23.1201171875 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda.out‚ 224.44921875 disk.rbd1.in‚0.046875 disk.rbd0.out‚0& disk.nvme0n1.out‚46788.36767578125) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17 network.sent‚ 4.383177e+06 network.recv‚ 1.0172e+06 disk.sda2.in‚0 disk.sda1.out‚0žú•2’ - _wandbruntime‚3961  _runtime‚3961@·É“ à ' lr-AdamW/pg2‚0.00026709227488362744 trainer/global_step‚2399 % _timestamp‚1.7843237084880288e+09 _runtime‚3961.862727504 _step‚193 ' lr-AdamW/pg1‚0.000267092274883627447±UΔ È % _timestamp‚1.7843237084880288e+09 _runtime‚3961.862727504 _step‚193 ' lr-AdamW/pg1‚0.00026709227488362744 ' lr-AdamW/pg2‚0.00026709227488362744 trainer/global_step‚2399Át_2• - _wandbruntime‚3961  _runtime‚3961Ë•è<ï'– éO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12943623 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚686429161684992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12943882 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2247821865876 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2251672856064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20737274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2253018245632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688682179930624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20737274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686429135511552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688381492583574 _step‚194 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249166466198 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186400059117568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6151989 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6791888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6151994 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚688680808367616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686132300689408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20358801 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2251672856064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20358801 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20358795 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚3961.864016868 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚686429135511552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2907063132160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688380122555284 trainer/global_step‚2399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20727301 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚686429161684992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6791634 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2907028595712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688680808367616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688682179930624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2253018245632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186399691026432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686132326117376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189307122249728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688380122555284 trainer/global_step‚2399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20727301 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚686429161684992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6791634 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2907028595712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688680808367616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688682179930624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2253018245632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186399691026432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686132326117376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189307122249728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2251672856064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20737274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2253018245632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688682179930624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.largæw‚"<e_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20737274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686429135511552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688381492583574 _step‚194 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249166466198 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186400059117568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6151989 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6791888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6151994 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚688680808367616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686132300689408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20358801 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2251672856064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20358801 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20358795 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚3961.864016868 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚686429135511552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ÂV¼Õ2˜ - _wandbruntime‚3963  _runtime‚3963ù™W¦ø&™ òM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2248745332122 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686410207895552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252597891584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686707136241664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20745766 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20367001 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252597891584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12949145 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189385395477504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250089932556 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 % _timestamp‚1.7843237098766727e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12948907 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚195 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186477545525248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚2399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688658927799706 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚686707162415104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚686707136241664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20745766 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2908218043392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20735779 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6794608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20367007 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2908252572672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688660297828108 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2253943288320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186477177434112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12948907 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚195 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186477545525248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚2399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688658927799706 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚686707162415104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚686707136241664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20745766 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2908218043392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20735779 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6794608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20367007 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2908252572672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688660297828108 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2253943288320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186477177434112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2248745332122 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686410207895552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252597891584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686707136241664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20745766 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20367001 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252597891584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12949145 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189385395477504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250089932556Ã"Ë…Ÿ2› - _wandbruntime‚3963  _runtime‚3963EØßMœ ü .loss/FBt1/detection_loss‚0.643048838009524 /loss/FRt1/detection_loss‚1.6100157931865797 1loss/SRqqt1/detection_loss‚0.7172398495612672 1loss/SRqqt2/detection_loss‚1.0592249410979653 epoch‚5 % _timestamp‚1.7843237099617157e+09 -loss/FRt1/assignment_loss‚1.0321198852615 2loss/SRqqt1/assignment_loss‚0.6550407082397911 &loss/total_loss‚10.513299482704245 _runtime‚3963.336114291 0loss/FRt2/assignment_loss‚2.0225198060385328 1loss/FBt1/assignment_loss‚0.06670287349655396 0loss/FBt2/assignment_loss‚0.1676502874413927 .loss/FBt2/detection_loss‚1.260750370612368 trainer/global_step‚2399 _step‚196 2loss/SRqqt2/assignment_loss‚0.6480605394645951 /loss/FRt2/detection_loss‚0.6309255902941737úÍ  epoch‚5 % _timestamp‚1.7843237099617157e+09 -loss/FRt1/assignment_loss‚1.0321198852615 2loss/SRqqt1/assignment_loss‚0.6550407082397911 &loss/total_loss‚10.513299482704245 _runtime‚3963.336114291 0loss/FRt2/assignment_loss‚2.0225198060385328 1loss/FBt1/assignment_loss‚0.06670287349655396 0loss/FBt2/assignment_loss‚0.1676502874413927 .loss/FBt2/detection_loss‚1.260750370612368 trainer/global_step‚2399 _step‚196 2loss/SRqqt2/assignment_loss‚0.6480605394645951 /loss/FRt2/detection_loss‚0.6309255902941737 .loss/FBt1/detection_loss‚0.643048838009524 /loss/FRt1/detection_loss‚1.6100157931865797 1loss/SRqqt1/detection_loss‚0.7172398495612672 1loss/SRqqt2/detection_loss‚1.0592249410979653ÄÍjs”ž ‚(:‰ ‰½êÒ«ëÖ¬ gpu.0.gpu‚77.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚335.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zãÞŠ+Ÿ ‚(:  ‰½êÒ»ÁÇ®! disk.nvme0n1.in‚ 8305.51953125 network.recv‚ 1.020587e+06' disk.nvme0c0n1.out‚46802.7080078125 disk.sda.in‚ 0.01171875 disk.sda2.in‚0# disk.rbd1.out‚17748.19580078125' proc.memory.availableMB‚ 1944.609375 network.sent‚ 4.387731e+06 disk.drbd2165.out‚0 disk.dm-1.out‚ 226.65625$ disk./.usageGB‚582.4727897644043 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 8305.51953125 proc.cpu.threads‚17 disk.sda3.out‚ 226.65625# disk.nvme0c0n1.in‚ 8305.51953125 disk.rbd2.out‚ 23.1201171875) disk./.usagePercent‚39.09023764831233 disk.rbd1.in‚0.046875$ memory_percent‚88.13104629516602# proc.memory.rssMB‚ 2884.27734375 cpu‚25.48886893908827 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875" disk.dm-0.out‚46802.7080078125 disk.sda1.in‚0) proc.memory.percent‚17.60423183441162% disk.nvme0n1.out‚46802.7080078125 disk.sda.out‚ 226.65625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625ÛÆü2  - _wandbruntime‚3989  _runtime‚3989Y•Ô$¡ Œ| 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5422570109367371 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23932474851608276 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42173463106155396 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18116149306297302 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.518835723400116 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2711339592933655 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4825846254825592 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5970452427864075 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025938430801033974 )loss/val_total_loss‚9.649814341228137 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45486053824424744 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49133923649787903 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 )jet/accuracy_1_of_1‚0.243143692612648 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18019522726535797 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4476325213909149 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5421122312545776 7"Purity/1FRt*SRqqt2FBt/event_purity‚0.62061607837677 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5505933165550232 +jet/accuracy_2_of_2‚0.11453073471784592 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8891580700874329 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1266779750585556 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5033333897590637 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6308887600898743 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849184155464172 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5990845561027527 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7310876250267029 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47433504462242126 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4430970251560211 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013557632453739643 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4138191342353821 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13225801289081573 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1991349160671234 *jet/accuracy_2_of_4‚0.6912516355514526 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 _runtime‚ 3989.41364147 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4391593039035797 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4935113489627838 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6935374140739441 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15776512026786804 *particle/specificity‚0.904575526714325 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.26459768414497375 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4964521825313568 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.19322864711284637 8 σ~ûî+Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45659318566322327 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6549904346466064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.718923032283783 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5546875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5321433544158936 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16284941136837006 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5359339714050293 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21313020586967468 *jet/accuracy_1_of_3‚0.8763537406921387 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1825169175863266 _step‚197 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22854246199131012 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8560633063316345 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.08650225400924683 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14283746480941772 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 ,jet/accuracy_3_of_3‚0.061543770134449005 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.019879024475812912 (particle/accuracy‚0.8367629051208496 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6103330254554749 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03046785295009613 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5564688444137573 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16907444596290588 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2114911824464798 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.4085460901260376 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8359236717224121 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18273992836475372 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25303560495376587 particle/accuracy_1_of_2‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/sensitivity‚0.5887855291366577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1973983496427536 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18978595733642578 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843237360381792e+09 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03814703971147537 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1921539604663849 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9831352233886719 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5535241365432739 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019726112484931946 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4597140848636627 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6518488526344299 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933008551597595 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5268554091453552 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8953335285186768 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6160035729408264 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16907444596290588 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6518488526344299 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4391593039035797 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955881834030151 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5374910831451416 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4928780496120453 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4483507573604584 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15776512026786804 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2571186423301697 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6923172473907471 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5944086313247681 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6899848580360413 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46968168020248413 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /particle/accuracy_4_of_4‚0.9174996614456177 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929463267326355 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13277336955070496 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951707720756531 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14328765869140625 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 epoch‚5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23673087358474731 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7043839693069458 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08826185762882233 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9965986609458923 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2165345698595047 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9955384135246277 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 'particle/f_score‚0.6076996922492981 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6850907206535339 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841141700744629 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2243681401014328 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5993333458900452 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1999092549085617 +jet/accuracy_3_of_4‚0.08050160109996796 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970422387123108 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5268570184707642 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5354273319244385 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906672239303589 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9419024586677551 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9864527583122253 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1782018393278122 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5334488153457642 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9943369626998901 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18080219626426697 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19684965908527374 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17018917202949524 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5946358442306519 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23719333112239838 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 trainer/global_step‚2399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4879290759563446 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9916711449623108 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13734963536262512 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_2‚0.6886363625526428 5validation_average_jet_accuracy‚0.290874183177948 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944986701011658 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6803099513053894 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_5_of_5‚0²´%Ã>¢ ‘| & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5354273319244385 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906672239303589 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9419024586677551 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9864527583122253 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1782018393278122 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5334488153457642 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9943369626998901 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18080219626426697 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19684965908527374 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17018917202949524 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5946358442306519 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23719333112239838 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 trainer/global_step‚2399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4879290759563446 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9916711449623108 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13734963536262512 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_2‚0.6886363625526428 5validation_average_jet_accuracy‚0.290874183177948 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944986701011658 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6803099513053894 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5422570109367371 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23932474851608276 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42173463106155396 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18116149306297302 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.518835723400116 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2711339592933655 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4825846254825592 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5970452427864075 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025938430801033974 )loss/val_total_loss‚9.649814341228137 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45486053824424744 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49133923649787903 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 )jet/accuracy_1_of_1‚0.243143692612648 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18019522726535797 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4476325213909149 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5421122312545776 7"Purity/1FRt*SRqqt2FBt/event_purity‚0.62061607837677 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5505933165550232 +jet/accuracy_2_of_2‚0.11453073471784592 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8891580700874329 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1266779750585556 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5033333897590637 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6308887600898743 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849184155464172 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5990845561027527 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7310876250267029 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47433504462242126 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4430970251560211 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013557632453739643 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4138191342353821 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13225801289081573 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1991349160671234 *jet/accuracy_2_of_4‚0.6912516355514526 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 _runtime‚ 3989.41364147 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4391593039035797 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4935113489627838 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6935374140739441 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15776512026786804 *particle/specificity‚0.904575526714325 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.26459768414497375 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4964521825313568 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.19322864711284637 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45659318566322327 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6549904346466064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.718923032283783 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5546875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5321433544158936 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16284941136837006 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5359339714050293 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21313020586967468 *jet/accuracy_1_of_3‚0.8763537406921387 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1825169175863266 _step‚197 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22854246199131012 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8560633063316345 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.08650225400924683 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14283746480941772 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 ,jet/accuracy_3_of_3‚0.061543770134449005 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.019879024475812912 (particle/accuracy‚0.8367629051208496 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6103330254554749 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03046785295009613 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5564688444137573 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16907444596290588 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2114911824464798 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.4085460901260376 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8359236717224121 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18273992836475372 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25303560495376587 particle/accuracy_1_of_2‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/sensitivity‚0.5887855291366577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1973983496427536 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18978595733642578 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843237360381792e+09 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03814703971147537 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1921539604663849 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9831352233886719 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5535241365432739 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019726112484931946 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4597140848636627 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6518488526344299 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933008551597595 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5268554091453552 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8953335285186768 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6160035729408264 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16907444596290588 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6518488526344299 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4391593039035797 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955881834030151 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5374910831451416 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4928780496120453 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4483507573604584 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15776512026786804 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2571186423301697 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6923172473907471 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5944086313247681 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6899848580360413 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46968168020248413 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /particle/accuracy_4_of_4‚0.9174996614456177 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929463267326355 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13277336955070496 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951707720756531 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14328765869140625 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 epoch‚5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23673087358474731 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7043839693069458 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08826185762882233 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9965986609458923 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2165345698595047 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9955384135246277 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 'particle/f_score‚0.6076996922492981 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6850907206535339 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841141700744629 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2243681401014328 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5993333458900452 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1999092549085617 +jet/accuracy_3_of_4‚0.08050160109996796 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970422387123108 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5268570184707642 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1ÅdŸçó“£ ‚(:ˆ ˜½êÒŽõª¬ gpu.0.gpu‚30.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê(ùÖ.¤ ‚(:£ ˜½êÒ£Ãý­) disk./.usagePercent‚39.09046907912329 disk.sda3.out‚ 228.72265625! proc.memory.rssMB‚ 3121.546875* proc.memory.percent‚19.052410125732422 network.recv‚ 1.025432e+06 disk.rbd0.out‚0' disk.nvme0c0n1.out‚46834.7158203125) proc.memory.availableMB‚ 1707.47265625 cpu‚25.49844535898307 disk.sda.in‚ 0.01171875$ memory_percent‚89.57841396331787 disk.sda2.in‚0 disk.rbd1.in‚0.046875 disk.rbd2.out‚ 23.1201171875" disk.dm-0.out‚46834.7158203125 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 8356.765625 disk.sda.out‚ 228.72265625# disk.rbd1.out‚17748.19580078125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 228.72265625 disk.sda3.in‚ 0.01171875$ disk./.usageGB‚582.4762382507324 disk.dm-0.in‚ 8356.765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 8356.765625% disk.nvme0n1.out‚46834.7158203125 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 network.sent‚ 4.45087e+06JƒM™“¥ ‚(:ˆ §½êÒåƒõ« gpu.0.gpu‚29.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ*;¦ ‚(:° §½êÒýÄ¥® disk.sda3.out‚ 231.44140625 disk.dm-0.in‚ 8357.59765625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875# proc.memory.rssMB‚ 2884.27734375 disk.sda.out‚ 231.44140625 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 4.508675e+06( disk.nvme0c0n1.out‚49757.35595703125& disk.nvme0n1.out‚49757.35595703125 disk.sda2.in‚0 disk.rbd0.in‚0$ memory_percent‚82.19199180603027 cpu‚25.490940350955913 network.recv‚ 1.030737e+06 disk.sda.in‚ 0.01171875 disk.rbd2.out‚ 23.1201171875) disk./.usagePercent‚39.09007943565617$ disk./.usageGB‚582.4704322814941# disk.nvme0c0n1.in‚ 8357.59765625 disk.rbd1.in‚ 0.15234375# disk.rbd1.out‚17748.36767578125) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17! disk.nvme0n1.in‚ 8357.59765625 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875# disk.dm-0.out‚49757.35595703125 disk.dm-1.out‚ 231.44140625( proc.memory.availableMB‚ 2917.6640625 disk.rbd2.in‚0.015625+TÁ:§ ‚(:’ ¶½êÒ¶üò­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚276.45' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.28999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GBY(8¨ ‚(:­ ¶½êÒ’úº®) proc.memory.availableMB‚ 2917.48828125# disk.rbd1.out‚17748.37939453125# disk.dm-0.out‚49775.98974609375) proc.memory.percent‚17.60423183441162 cpu‚25.490539204453302) disk./.usagePercent‚39.08967980789522# disk.nvme0c0n1.in‚ 8357.83984375 disk.sda.out‚ 233.62890625 disk.rbd2.out‚ 23.1201171875$ disk./.usageGB‚582.4644775390625 disk.drbd2165.out‚0 disk.rbd1.in‚0.984375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0! disk.nvme0n1.in‚ 8357.83984375 network.sent‚ 4.5143e+06 network.recv‚ 1.033929e+06 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 233.62890625 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875& disk.nvme0n1.out‚49775.98974609375 disk.sda1.in‚0# proc.memory.rssMB‚ 2884.27734375 disk.sda3.out‚ 233.62890625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚49775.98974609375 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 8357.83984375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚82.19306468963623FUk^”© ‚(:‰ ŽêÒ×ñå« gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚238.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ËÕ©½ª ‚(:Ÿ ŽêÒõƒ¦®$ disk./.usageGB‚582.4676322937012! disk.nvme0c0n1.in‚ 8358.796875 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 235.23828125$ memory_percent‚82.19575881958008# proc.memory.rssMB‚ 2884.27734375) disk./.usagePercent‚39.08989152612604 disk.sda.in‚ 0.01171875 disk.sda2.out‚0" disk.rbd1.out‚17748.5595703125 disk.rbd0.out‚0 disk.sda1.in‚0" disk.dm-0.out‚49785.4951171875 disk.sda3.out‚ 235.23828125' disk.nvme0c0n1.out‚49785.4951171875 disk.rbd1.in‚4.375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 23.1201171875 network.sent‚ 4.518835e+06 disk.sda3.in‚ 0.01171875 disk.sda2.in‚0' pã-ï~mroc.memory.availableMB‚ 2917.046875 network.recv‚ 1.037753e+06% disk.nvme0n1.out‚49785.4951171875 disk.sda.out‚ 235.23828125 disk.dm-0.in‚ 8358.796875 cpu‚25.49020667424306 disk.nvme0n1.in‚ 8358.796875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17¬1á7« ‚(:¬ Ô½êÒŒ¹ƒ¯ disk.sda1.out‚0& disk.nvme0n1.out‚49810.57373046875 disk.sda2.out‚0) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17 network.recv‚ 1.041079e+06 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 cpu‚25.48974266754373" disk.rbd1.out‚17748.5595703125 disk.rbd0.out‚0 disk.rbd2.out‚ 23.1201171875$ disk./.usageGB‚582.4692459106445 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda.out‚ 238.05078125 disk.sda2.in‚0# proc.memory.rssMB‚ 2884.27734375" disk.nvme0c0n1.in‚ 8437.0078125 disk.nvme0n1.in‚ 8437.0078125$ memory_percent‚82.19621181488037( disk.nvme0c0n1.out‚49810.57373046875 network.sent‚ 4.524479e+06 disk.rbd0.in‚0) disk./.usagePercent‚39.08999981731303 disk.sda1.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 8437.0078125# disk.dm-0.out‚49810.57373046875 disk.rbd1.in‚ 9.04296875 disk.sda3.out‚ 238.05078125 disk.dm-1.out‚ 238.05078125) proc.memory.availableMB‚ 2916.97265625bàÐ)Ÿ¬ ‚(:” Ô½êÒ´Ú÷® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚298.615' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.723000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÖñ~Ÿ­ ‚(:” ã½êÒÄ”†¬ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚258.899' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.779799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]CH¡)® ‚(:ž ã½êÒúàή# proc.memory.rssMB‚ 2884.27734375 network.recv‚ 1.044475e+06) disk./.usagePercent‚39.09053359302192 disk.sda3.in‚ 0.01171875$ disk.nvme0c0n1.out‚ 49833.4296875 disk.rbd0.out‚0 cpu‚25.48941862770652 disk.sda2.out‚0 disk.dm-0.in‚ 8437.4453125 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚582.4771995544434 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 240.27734375 disk.sda2.in‚0 disk.rbd1.in‚ 11.87890625 disk.rbd2.out‚ 23.1201171875' proc.memory.availableMB‚ 2917.453125 disk.sda3.out‚ 240.27734375" disk.nvme0c0n1.in‚ 8437.4453125 network.sent‚ 4.528997e+06 disk.rbd2.in‚0.015625$ memory_percent‚82.19327926635742) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17! disk.rbd1.out‚17748.716796875 disk.rbd0.in‚0 disk.dm-0.out‚ 49833.4296875 disk.dm-1.out‚ 240.27734375 disk.nvme0n1.in‚ 8437.4453125" disk.nvme0n1.out‚ 49833.4296875 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd2165.in‚0<²c2¯ - _wandbruntime‚4072  _runtime‚4072‚O³bÉ° à _runtime‚4072.228365194 _step‚198 ' lr-AdamW/pg1‚0.00026674221056236256 ' lr-AdamW/pg2‚0.00026674221056236256 trainer/global_step‚2449 % _timestamp‚1.7843238188537254e+09:b%Vα È ' lr-AdamW/pg2‚0.00026674221056236256 trainer/global_step‚2449 % _timestamp‚1.7843238188537254e+09 _runtime‚4072.228365194 _step‚198 ' lr-AdamW/pg1‚0.00026674221056236256ÆÃWA…2² - _wandbruntime‚4072  _runtime‚4072…BŠ ï'³ éO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2299888110426 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20901107 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706462356416000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190322491748352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚706153556669532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305176790528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21222013 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13252002 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13251747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704157153452032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚2449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20901107 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚706460984852992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974558292480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21222013 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704157179625472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21212040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20901101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704157179625472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706460984852992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706152186641242 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193296681949696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6288062 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚703852323958784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706462356416000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119Gœ ô'´ îO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2299888110426 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20901107 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706462356416000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190322491748352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚706153556669532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305176790528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21222013 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13252002 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13251747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704157153452032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚2449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20901107 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚706460984852992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974558292480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21222013 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704157179625472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21212040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20901101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704157179625472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706460984852992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706152186641242 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193296681949696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6288062 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚703852323958784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706462356416000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736ÇèK2µ - _wandbruntime‚4073  _runtime‚4073Pè\“Š ¶ òM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975759093248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6290600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42129825 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704435180355584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190399978156032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193375369158144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784323820255356e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚704130180308992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚706741282188800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20909307 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302156177106 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21230505 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6290605 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20909313 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4073.630362765 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21220518 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704435154182144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2304756436480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13257065 _KDeviceStatsMoN`ãnnitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190399610064896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706741282188800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚706432361914066 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20909313 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _step‚200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0{`;ßý&· ÷M J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21220518 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704435154182144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2304756436480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13257065 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190399610064896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706741282188800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚706432361914066 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20909313 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _step‚200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975759093248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6290600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42129825 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704435180355584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190399978156032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193375369158144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784323820255356e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚704130180308992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚706741282188800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20909307 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302156177106 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21230505 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6290605 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20909313 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4073.630362765Èjió2¸ - _wandbruntime‚4073  _runtime‚4073Cǥɹ ú epoch‚6 _step‚201 2loss/SRqqt1/assignment_loss‚0.5724124138051181 2loss/SRqqt2/assignment_loss‚0.6309116663345996 0loss/FBt2/assignment_loss‚0.2836395754825968 .loss/FRt1/detection_loss‚1.629893950818152 1loss/SRqqt1/detection_loss‚0.6880777056489756 trainer/global_step‚2449 $ _timestamp‚1.784323820335209e+09 0loss/FRt1/assignment_loss‚0.8683097241855249 /loss/FRt2/assignment_loss‚2.020039688267645 2loss/FBt1/assignment_loss‚0.049142177227089204 /loss/FBt1/detection_loss‚0.5687946817044922 %loss/total_loss‚9.948561766350036 ,loss/FRt2/detection_loss‚0.6442518803472 0loss/SRqqt2/detection_loss‚1.017690918734454 /loss/FBt2/detection_loss‚0.9753973837941885 _runtime‚4073.709785423´¯Tº ÿ %loss/total_loss‚9.948561766350036 ,loss/FRt2/detection_loss‚0.6442518803472 0loss/SRqqt2/detection_loss‚1.017690918734454 /loss/FBt2/detection_loss‚0.9753973837941885 _runtime‚4073.709785423 epoch‚6 _step‚201 2loss/SRqqt1/assignment_loss‚0.5724124138051181 2loss/SRqqt2/assignment_loss‚0.6309116663345996 0loss/FBt2/assignment_loss‚0.2836395754825968 .loss/FRt1/detection_loss‚1.629893950818152 1loss/SRqqt1/detection_loss‚0.6880777056489756 trainer/global_step‚2449 $ _timestamp‚1.784323820335209e+09 0loss/FRt1/assignment_loss‚0.8683097241855249 /loss/FRt2/assignment_loss‚2.020039688267645 2loss/FBt1/assignment_loss‚0.049142177227089204 /loss/FBt1/detection_loss‚0.5687946817044922ÉŒqË$Ÿ» ‚(:” ò½êÒëÊÅ« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚290.708' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.141600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É¢wß ¼ ‚(:• ò½êÒ‹’×­# disk.nvme0c0n1.in‚ 8438.40234375 disk.nvme0n1.out‚ 49848.125) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17 disk.sda1.in‚0! disk.nvme0n1.in‚ 8438.40234375 disk.rbd0.out‚0$ memory_percent‚82.19513893127441 network.sent‚ 4.534612e+06 disk.drbd2165.in‚0 disk.nvme0c0n1.out‚ 49848.125 disk.sda3.out‚ 242.12109375# disk./.usageGB‚582.468807220459 disk.dm-0.in‚ 8438.40234375 disk.sda1.out‚0 disk.rbd1.in‚ 15.7421875 disk.rbd0.in‚0 disk.rbd2.out‚ 23.1201171875 network.recv‚ 1.047653e+06) disk./.usagePercent‚39.08997037644659 disk.sda.in‚ 0.01171875 disk.sda.out‚ 242.12109375 disk.sda2.out‚0! disk.rbd1.out‚17748.716796875( proc.memory.availableMB‚ 2917.1484375 disk.dm-0.out‚ 49848.125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2884.27734375 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 242.12109375 cpu‚25.488982343098915Ýï ž½ ‚(:“ ¾êÒßëû« gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚335.896' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.17920000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›SA86¾ ‚(:« ¾êÒÊõê® disk.sda1.out‚0 disk.sda3.in‚ 0.01171875# disk.rbd1.out‚17748.80322265625# disk.dm-0.out‚49884.88916015625 network.sent‚ 4.597855e+06 disk.sda3.out‚ 244.75390625 network.recv‚ 1.052912e+06( disk.nvme0c0n1.out‚49884.88916015625 disk.nvme0n1.in‚ 8438.421875 disk.rbd0.out‚0) proc.memory.availableMB‚ 2917.09765625# proc.memory.rssMB‚ 2884.27734375) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17 disk.dm-1.out‚ 244.75390625 disk.sda.in‚ 0.01171875 disk.sda.out‚ 244.75390625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625) disk./.usagePercent‚39.10434340744144! disk.nvme0c0n1.in‚ 8438.421875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 18.87890625 disk.rbd2.out‚ 23.1201171875 cpu‚25.488703871300515# disk./.usageGB‚582.682975769043 disk.drbd2165.out‚0& disk.nvme0n1.out‚49884.88916015625 disk.dm-0.in‚ 8438.421875$ memory_percent‚82.19544887542725 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda2.out‚0\§¿ ‚(:’ ¾êÒÝ×ä¬ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚271.89' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.37799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ø15À ‚(:ª ¾êÒĹ­¯ disk.sda3.out‚ 246.5234375# disk.nvme0c0n1.in‚ 8484.82421875 disk.rbd2.out‚ 23.1201171875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0# disk./.usageGB‚582.795783996582# disk.rbd1.out‚17748.80322265625 disk.dm-0.in‚ 8484.82421875 disk.dm-1.out‚ 246.5234375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2884.27734375 cpu‚25.48837565938034 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.01171875 disk.sda2.in‚0 network.sent‚ 4.602364e+06) disk./.usagePercent‚39.11191406224399 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚50069.02294921875 disk.rbd0.in‚0 network.recv‚ 1.056297e+06 disk.sda.out‚ 246.5234375# disk.dm-0.out‚50069.02294921875) proc.memory.percent‚17.60423183441162& disk.nvme0n1.out‚50069.02294921875 disk.sda.in‚ 0.01171875! disk.nvme0n1.in‚ 8484.82421875 disk.rbd1.in‚ 22.62890625$ memory_percent‚82.19547271728516& proc.memory.availableMB‚ 2917.09375 proc.cpu.threads‚17}ôb”Á ‚(:‰ Ÿ¾êңݚ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á¶à2 ‚(:§ Ÿ¾êҶЮ disk.nvme0n1.in‚ 8485.78125# disk.dm-0.out‚50094.40185546875 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚82.18305110931396 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd2.in‚0.015625& disk.nvme0n1.out‚50094.40185546875 disk.rbd1.in‚ 22.62890625 disk.rbd2.out‚ 23.1201171875 disk.dm-0.in‚ 8485.78125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.60423183441162 disk.sda.out‚ 247.1328125 disk.sda2.out‚0 disk.sda3.in‚ 0.01171875 disk.nvme0c0n1.in‚ 8485.78125 network.recv‚ 1.059981e+06 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 247.1328125# proc.memory.rssMB‚ 2884.27734375* disk./.usagePercent‚39.225932137768595$ disk./.usageGB‚584.4947357177734# disk.rbd1.out‚17748.89697265625 disk.dm-1.out‚ 247.1328125( disk.nvme0c0n1.out‚50094.40185546875) proc.memory.availableMB‚ 2919.12890625 cpu‚25.487951304473686 network.sent‚ 4.607579e+06î2ƒãœà ‚(:‘ ®¾êÒ„à⬠gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚325.692' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.13839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E H&4Ä ‚(:© ®¾êÒðÛ­$ disk.nvme0n1.out‚50322.298828125# proc.memory.rssMB‚ 2884.27734375 disk.sda.out‚ 248.9765625 network.recv‚ 1.063178e+06 disk.sda3.out‚ 248.9765625 disk.sda.in‚ 0.01171875 cpu‚25.487639706803527 proc.cpu.threads‚17) disk./.usagePercent‚39.30128232330647 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 248.9765625$ disk./.usageGB‚585.6175079345703# disk.nvme0c0n1.in‚ 8485.78515625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 8485.78515625! disk.dm-0.out‚50322.298828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.60423183441162& disk.nvme0c0n1.out‚50322.298828125 disk.rbd1.in‚ 22.62890625# disk.rbd1.out‚17748.89697265625! disk.nvme0n1.in‚ 8485.78515625) proc.memory.availableMB‚ 2918.87109375 disk.drbd2165.out‚0 network.sent‚ 4.612647e+06 disk.sda3.in‚ 0.01171875 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 23.1201171875$ memory_percent‚82.18462467193604Áâî"2Å - _wandbruntime‚4152  _runtime‚4152U©„™ÇÆ Á trainer/global_step‚2499 % _timestamp‚1.7843238990425622e+09 _runtime‚4152.418520118 _step‚202 & lr-AdamW/pg1‚0.0002663838152740399 & lr-AdamW/pg2‚0.0002663838152740399]¼`¶ÌÇ Æ & lr-AdamW/pg2‚0.0002663838152740399 trainer/global_step‚2499 % _timestamp‚1.7843238990425622e+09 _runtime‚4152.418520118 _step‚202 & lr-AdamW/pg1‚0.0002663838152740399ÊC¢\2È - _wandbruntime‚4152  _runtime‚4152ý>„7ï'É éO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718057216131072 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 % _timestamp‚1.7843238990433488e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21647765 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2350147331584 _step‚203 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2351492721152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21311407 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717746412867584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13518338 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21311401 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7103414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2350147331584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103131 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2499 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194196812132352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718057189957632 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720408708852224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21647765 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194196444041216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720408708852224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13518626 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6415207 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3034413103616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3034376470016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21311401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21637792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197231225235968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2346125223534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197230820511232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351492721152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21311407 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720407337289216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718057189957632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2347469823856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 % _timestamp‚1.7843238990433488e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21647765 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2350147331584 _step‚203 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2351492721152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21311407 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717746412867584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13518338 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21311401 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7103414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2350147331584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103131 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2499 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194196812132352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718057189957632 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720408708852224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21647765 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194196444041216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720408708852224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13518626 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6415207 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3034413103616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3034376470016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21311401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21637792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197231225235968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2346125223534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197230820511232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351492721152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21311407 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720407337289216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718057189957632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2347469823856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21646270 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3035601165824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2348393290214 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352417763840 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21319607 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6417750 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7105888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3035564539392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21319613 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720687634625024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21319613 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13523911 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2351072367104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21656257 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _step‚204 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720687634625024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197309899705856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718335190687744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718335190687744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6417755 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2347048689780 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720372687935974 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 $ _timestamp‚1.784323900439587e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13523638 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194274298540032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194273930448896 _runtime‚ 4153.81449604 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2352417763840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718335190687744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718335190687744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6417755 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2347048689780 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720372687935974 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 $ _timestamp‚1.784323900439587e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13523638 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194274298540032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194273930448896 _runtime‚ 4153.81449604 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2352417763840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21646270 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3035601165824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2348393290214 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352417763840 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21319607 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6417750 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7105888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3035564539392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21319613 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720687634625024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21319613 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13523911 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2351072367104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21656257 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _step‚204 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720687634625024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197309899705856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820Ì{óðÛ2Î - _wandbruntime‚4153  _runtime‚4153­ ]Ï û /loss/FRt1/detection_loss‚1.5168252672047846 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚0.5585838870655307 %loss/total_loss‚9.801841421472519 _step‚205 epoch‚6 /loss/FRt1/assignment_loss‚0.978285636392872 1loss/SRqqt1/detection_loss‚0.7289561328008459 /loss/FBt1/detection_loss‚0.5348360583104542 $ _timestamp‚1.784323900520755e+09 _runtime‚ 4153.89540384 .loss/FRt2/detection_loss‚0.663547549465395 1loss/SRqqt2/detection_loss‚0.9606220490646129 /loss/FBt2/detection_loss‚0.8470331347823419 /loss/FRt2/assignment_loss‚2.077155854289436 2loss/SRqqt2/assignment_loss‚0.7007534591408996 0loss/FBt1/assignment_loss‚0.0845979834971575 1loss/FBt2/assignment_loss‚0.15064440945818827ûƒ^Ð € 1loss/FBt2/assignment_loss‚0.15064440945818827 /loss/FRt1/detection_loss‚1.5168252672047846 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚0.5585838870655307 %loss/total_loss‚9.801841421472519 _step‚205 epoch‚6 /loss/FRt1/assignment_loss‚0.978285636392872 1loss/SRqqt1/detection_loss‚0.7289561328008459 /loss/FBt1/detection_loss‚0.5348360583104542 $ _timestamp‚1.784323900520755e+09 _runtime‚ 4153.89540384 .loss/FRt2/detection_loss‚0.663547549465395 1loss/SRqqt2/detection_loss‚0.9606220490646129 /loss/FBt2/detection_loss‚0.8470331347823419 /loss/FRt2/assignment_loss‚2.077155854289436 2loss/SRqqt2/assignment_loss‚0.7007534591408996 0loss/FBt1/assignment_loss‚0.0845979834971575Í;¤”Ñ ‚(:‰ ½¾êÒ͔Ǯ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚270.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à{H‘1Ò ‚(:¦ ½¾êÒ½Ìñ¯ disk.dm-1.out‚ 250.609375' proc.memory.availableMB‚ 2918.640625# disk.dm-0.out‚52758.33544921875 disk.drbd2165.in‚0$ memory_percent‚82.18603134155273 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 network.recv‚ 1.066712e+06 disk.sda.out‚ 250.609375 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚29.8125# disk.rbd1.out‚17749.13916015625 disk.dm-0.in‚ 8486.62109375# proc.memory.rssMB‚ 2884.27734375! disk.nvme0n1.in‚ 8486.62109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 4.617704e+06) disk./.usagePercent‚39.28206281368091 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 8486.62109375( disk.nvme0c0n1.out‚52758.33544921875) proc.memory.percent‚17.60423183441162 cpu‚25.487237874018824 disk.sda3.out‚ 250.609375$ disk./.usageGB‚585.3311233520508& disk.nvme0n1.out‚52758.33544921875 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 23.1201171875›Œå¦žÓ ‚(:“ ̾êÒæ¶ý­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚292.65' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.529999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý4'èÔ ‚(:£ ̾êÒ«÷Ю disk.rbd1.in‚ 51.73046875 disk.sda1.in‚0$ disk./.usageGB‚585.3447227478027 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.01171875 disk.sda2.out‚0# disk.rbd1.out‚1DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198070764425216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚731957252636672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2396463262208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21721707 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731640501776384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7242578 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22073517 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734353689725440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201164514298368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734034208713348 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22073517 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6542357 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198071132516352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚734032838685058 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2396463262208 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731640476348416 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚207 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22063544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2397808651776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2393706936964 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21721701ƒ¼ûsô'â îO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734353689725440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201164514298368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734034208713348 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22073517 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6542357 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198071132516352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚734032838685058 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2396463262208 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731640476348416 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚207 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22063544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2397808651776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2393706936964 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21721701 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198070764425216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚731957252636672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2396463262208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21721707 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731640501776384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7242578 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22073517 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2397388297728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21729907 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13789975 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2398733694464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7245075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3094967525888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3094930899456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198148250832896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201243586449920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734633987061248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2394630403322 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2398733694464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22082009 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732235253366784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22072022 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13790240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734632615491072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on"’þa _train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21729913 _runtime‚4233.945802819 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6544905 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201243181732352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732235253366784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734313013957882 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734311643929480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2397388297728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2393285802888 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚731918358126592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732235227193344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 trainer/global_step‚2549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21729907 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21729913 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13789975 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2398733694464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7245075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3094967525888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3094930899456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198148250832896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201243586449920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734633987061248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2394630403322 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2398733694464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22082009 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732235253366784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22072022 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13790240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734632615491072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21729913 _runtime‚4233.945802819 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6544905 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201243181732352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732235253366784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734313013957882 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734311643929480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2397388297728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2393285802888 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚731918358126592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732235227193344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 trainer/global_step‚2549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21729907 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21729913 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2397388297728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648з¢`t2æ - _wandbruntime‚4234  _runtime‚4234^dTûç þ 1loss/FBt2/assignment_loss‚0.20230293149745648 0loss/SRqqt1/detection_loss‚0.667072894997952 0loss/SRqqt2/detection_loss‚1.052834442439195 _step‚209 0loss/FRt1/assignment_loss‚0.9507668935225911 1loss/SRqqt2/assignment_loss‚0.695669583000662 /loss/FRt1/detection_loss‚1.5507690573989026 0loss/FBt1/detection_loss‚0.43962001886915786 %loss/total_loss‚9.844697980325847 epoch‚6 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚0.5814283168169035 1loss/FBt1/assignment_loss‚0.08158526681111038 .loss/FRt2/detection_loss‚0.634937949860492 % _timestamp‚1.7843239806521487e+09 0loss/FRt2/assignment_loss‚2.0825266960927484 /loss/FBt2/detection_loss‚0.9051839290186763 _runtime‚4234.026706015 ®w› è ƒ 0loss/FRt2/assignment_loss‚2.0825266960927484 /loss/FBt2/detection_loss‚0.9051839290186763 _runtime‚4234.026706015 1loss/FBt2/assignment_loss‚0.20230293149745648 0loss/SRqqt1/detection_loss‚0.667072894997952 0loss/SRqqt2/detection_loss‚1.052834442439195 _step‚209 0loss/FRt1/assignment_loss‚0.9507668935225911 1loss/SRqqt2/assignment_loss‚0.695669583000662 /loss/FRt1/detection_loss‚1.5507690573989026 0loss/FBt1/detection_loss‚0.43962001886915786 %loss/total_loss‚9.844697980325847 epoch‚6 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚0.5814283168169035 1loss/FBt1/assignment_loss‚0.08158526681111038 .loss/FRt2/detection_loss‚0.634937949860492 % _timestamp‚1.7843239806521487e+09Ѷ ‰±”é ‚(:‰ —¿êÒœš¬® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚286.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¼ 3ê ‚(:¨ —¿êÒ¯óç®" disk.nvme0c0n1.in‚ 8750.8984375# proc.memory.rssMB‚ 2884.27734375" disk.dm-0.out‚53359.1728515625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd2.in‚0.015625 network.recv‚ 1.08934e+06% disk.nvme0n1.out‚53359.1728515625 disk.rbd0.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1946.8046875 proc.cpu.threads‚17( disk./.usagePercent‚39.2994283167435 disk.nvme0n1.in‚ 8750.8984375 disk.rbd2.out‚ 23.1201171875 cpu‚25.48535380420619 disk.rbd1.in‚ 164.35546875 disk.sda3.in‚ 0.01171875) proc.memory.percent‚17.60423183441162$ disk./.usageGB‚585.5898818969727 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 263.29296875 network.sent‚ 4.706887e+06 disk.dm-0.in‚ 8750.8984375 disk.dm-1.in‚ 0.01171875' disk.nvme0c0n1.out‚53359.1728515625$ memory_percent‚88.11764717102051 disk.sda.out‚ 263.29296875" disk.rbd1.out‚17750.7568359375 disk.rbd0.out‚0 disk.sda3.out‚ 263.29296875%ôüñ”ë ‚(:‰ ¦¿êÒªÀ“¬ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚331.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E‰Ü÷'ì ‚(:œ ¦¿êÒæ㤮 disk.sda2.out‚0 disk.rbd2.in‚0.015625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚265.0625 network.sent‚ 4.769464e+06 network.recv‚ 1.094451e+06 disk.sda.out‚265.0625 disk.rbd1.in‚ 182.35546875 disk.dm-0.in‚ 8750.8984375 disk.sda3.in‚ 0.01171875 disk.sda2.in‚0) proc.memory.percent‚17.60423183441162) disk./.usagePercent‚39.29912827591337" disk.rbd1.out‚17750.7568359375 disk.rbd0.in‚0$ memory_percent‚88.12062740325928# proc.memory.rssMB‚ 2884.27734375 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1946.31640625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.01171875! disk.dm-0.out‚53381.931640625 disk.sda1.in‚0 disk.nvme0n1.in‚ 8750.8984375$ disk./.usageGB‚585.5854110717773 disk.dm-1.out‚265.0625& disk.nvme0c0n1.out‚53381.931640625 disk.sda.in‚ 0.01171875 disk.rbd2.out‚ 23.1201171875 cpu‚25.48506158748083" disk.nvme0c0n1.in‚ 8750.8984375$ disk.nvme0n1.out‚53381.931640625]–¯9”í ‚(:‰ µ¿êÒ¹çç¬ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚263.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lñ¶3î ‚(:¨ µ¿êÒ¨Úˆ¯! disk.nvme0n1.in‚ 8751.85546875 disk.rbd0.out‚0 disk.sda3.out‚ 266.6015625 disk.sda2.out‚0 cpu‚25.484696976692767 disk.sda1.out‚0 disk.sda.out‚ 266.6015625 disk.dm-1.in‚ 0.01171875' disk.nvme0c0n1.out‚53390.9130859375% disk.nvme0n1.out‚53390.9130859375 disk.rbd2.in‚0.015625) proc.memory.percent‚17.60423183441162) disk./.usagePercent‚39.29878522581749" disk.rbd1.out‚17750.8974609375 disk.sda.in‚ 0.01171875 disk.dm-1.out‚ 266.6015625 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 8751.85546875 disk.rbd0.in‚0( proc.memory.availableMB‚ 1947.0390625# proc.memory.rssMB‚ 2884.27734375 network.sent‚ 4.77509e+06$ disk./.usageGB‚585.5802993774414 disk.rbd2.out‚ 23.1201171875" disk.dm-0.out‚53390.9130859375 disk.sda1.in‚0 disk.drbd2165.in‚0# memory_percent‚88.1162166595459 proc.cpu.threads‚17 disk.dm-0.in‚ 8751.85546875 disk.drbd2165.out‚0 network.recv‚ 1.098076e+06 disk.sda2.in‚0 disk.rbd1.in‚ 207.4765625ó§Ï”ï ‚(:‰ Ä¿êÒ·ó”¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚297.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü™Ø,ð ‚(:¡ Ä¿êÒ½ïö­) disk./.usagePercent‚39.29879418608119 disk.drbd2165.in‚0" disk.rbd1.out‚17750.8974609375 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1946.796875# proc.memory.rssMB‚ 2884.27734375) proc.memory.percent‚17.60423183441162 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚53405.93701171875 disk.rbd1.in‚ 226.69921875 network.recv‚ 1.100704e+06 disk.sda3.in‚ 0.01171875 disk.sda.out‚ 267.7109375 disk.sda3.out‚ 267.7109375 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 23.9755859375 disk.dm-0.in‚ 8752.6875# disk.dm-0.out‚53405.93701171875$ disk./.usageGB‚585.5804328918457 disk.nvme0n1.in‚ 8752.6875& disk.nvme0n1.out‚53405.93701171875 network.sent‚ 4.779435e+06 disk.sda1.out‚0 disk.dm-1.out‚ 267.7109375 disk.nvme0c0n1.in‚ 8752.6875 disk.sda.in‚ 0.01171875 disk.sda2.in‚0 cpu‚25.484300806955222 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda2.out‚0$ memory_percent‚88.11769485473633 proc.cpu.threads‚17>E‡”ñ ‚(:‰ Ó¿êÒ€èõ¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚243.236' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.6472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Þß2ò ‚(:§ Ó¿êқȀ¯ disk.dm-0.in‚ 9000.72265625 disk.sda3.out‚ 269.95703125# disk.nvme0c0n1.in‚ 9000.72265625% disk.nvme0c0n1.out‚53426.97265625 disk.rbd0.out‚0 proc.cpu.threads‚17# disk./.usageGB‚585.682071685791 disk.sda.out‚ 269.95703125$ memory_percent‚88.11774253845215 network.sent‚ 4.78395e+06* disk./.usagePercent‚39.305615250823486 cpu‚25.483995404854277 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875# disk.nvme0n1.out‚53426.97265625" disk.rbd1.out‚17751.0849609375 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 23.9755859375( proc.memory.availableMB‚ 1946.7890625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 9000.72265625 disk.sda.in‚ 0.01171875 disk.rbd1.in‚ 248.26171875# proc.memory.rssMB‚ 2884.27734375 network.recv‚ 1.104124e+06 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.60423183441162 disk.dm-0.out‚53426.97265625 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 269.95703125 disk.sda2.in‚0 disk.rbd0.in‚0¾²,ž2ó - _wandbruntime‚4312  _runtime‚4312rc)½Æô À & lr-AdamW/pg1‚0.0002656421244060639 & lr-AdamW/pg2‚0.0002656421244060639 trainer/global_step‚2599 $ _timestamp‚1.784324058978204e+09 _runtime‚4312.352764938 _step‚210óȤ¨Ëõ Å $ _timestamp‚1.784324058978204e+09 _runtime‚4312.352764938 _step‚210 & lr-AdamW/pg1‚0.0002656421244060639 & lr-AdamW/pg2‚0.0002656421244060639 trainer/global_step‚2599Ò"+‡A2ö - _wandbruntime‚4312  _runtime‚4312¤øÐk÷ éO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2444124582400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2439944050072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22499269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748300042161664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843240589792879e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745857262968832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745857289142272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748300042161664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745534565257216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22132007 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚745857289142272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22489296 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22132001 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3153408180224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚745857262968832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205098458452992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748301413724672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22499269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6669512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6669507 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4312.354035747 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745534590685184 _step‚211 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22132007 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/globaEY)¾„l_step‚2599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748301413724672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2444124582400 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7381381 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201945452900352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14051148 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22132001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442779192832 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747974534735256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201945084809216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ë½ ô'ø îO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6669507 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4312.354035747 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745534590685184 _step‚211 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22132007 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚2599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748301413724672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2444124582400 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7381381 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201945452900352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14051148 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22132001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442779192832 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747974534735256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201945084809216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2444124582400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2439944050072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22499269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748300042161664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843240589792879e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745857262968832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745857289142272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748300042161664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745534565257216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22132007 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚745857289142272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22489296 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22132001 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3153408180224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚745857262968832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205098458452992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748301413724672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22499269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6669512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448ÓŒjè2ù - _wandbruntime‚4313  _runtime‚4313µÕØÌù&ú óM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14056177 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6672050 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚748251969951388 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4313.754446252 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843240603795125e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746135263698944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205177538321408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205177135700992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22140207 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3154564484096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚2599 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443704228352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748580339497472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _step‚212 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2443704228352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202022939308032O;Jþ&û øM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14056177 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6672050 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚748251969951388 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4313.754446252 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843240603795125e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746135263698944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205177538321408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205177135700992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22140207 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3154564484096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚2599 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443704228352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748580339497472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _step‚212 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2443704228352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202022939308032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987Ô‹Á72ü - _wandbruntime‚4313  _runtime‚4313_á×üý € %loss/total_loss‚9.427004986808175 1loss/FBt1/assignment_loss‚0.08552078214911077 .loss/FRt1/detection_loss‚1.453472921574351 epoch‚6 trainer/global_step‚2599 % _timestamp‚1.7843240604592996e+09 _step‚213 2loss/SRyiø^ qqt2/assignment_loss‚0.6043161695381242 /loss/FRt2/detection_loss‚0.6556669160043159 /loss/FBt2/detection_loss‚0.8228423582209695 0loss/FRt1/assignment_loss‚0.9063245205011236 2loss/SRqqt1/assignment_loss‚0.6173957454410687 1loss/SRqqt1/detection_loss‚0.7280264570118156 _runtime‚4313.833875107 0loss/FRt2/assignment_loss‚2.0260687373337265 1loss/FBt2/assignment_loss‚0.07716758453301417 1loss/SRqqt2/detection_loss‚0.8860058787257733 /loss/FBt1/detection_loss‚0.5641969157747824 à7 þ … 0loss/FRt1/assignment_loss‚0.9063245205011236 2loss/SRqqt1/assignment_loss‚0.6173957454410687 1loss/SRqqt1/detection_loss‚0.7280264570118156 _runtime‚4313.833875107 0loss/FRt2/assignment_loss‚2.0260687373337265 1loss/FBt2/assignment_loss‚0.07716758453301417 1loss/SRqqt2/detection_loss‚0.8860058787257733 /loss/FBt1/detection_loss‚0.5641969157747824 %loss/total_loss‚9.427004986808175 1loss/FBt1/assignment_loss‚0.08552078214911077 .loss/FRt1/detection_loss‚1.453472921574351 epoch‚6 trainer/global_step‚2599 % _timestamp‚1.7843240604592996e+09 _step‚213 2loss/SRqqt2/assignment_loss‚0.6043161695381242 /loss/FRt2/detection_loss‚0.6556669160043159 /loss/FBt2/detection_loss‚0.8228423582209695Õ˜»Ržÿ ‚(:“ â¿êÒÅ·¬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÔ^3€‚(:¨ â¿êÒ·ù÷®) proc.memory.percent‚17.60423183441162 disk.sda.in‚0.015625 disk.sda.out‚ 270.87109375 disk.drbd2165.in‚0 disk.sda3.out‚ 270.87109375 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 9001.6796875 disk.dm-1.out‚ 270.87109375$ memory_percent‚88.11945915222168) disk./.usagePercent‚39.43033828139234 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0.015625 proc.cpu.threads‚17 network.sent‚ 4.789039e+06 network.recv‚ 1.107662e+06 disk.nvme0n1.in‚ 9001.6796875" disk.rbd1.out‚17751.0849609375 disk.rbd0.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1946.5078125$ disk./.usageGB‚587.5405349731445& disk.nvme0n1.out‚53999.47021484375 disk.rbd1.in‚ 264.0234375 disk.rbd2.out‚24.09423828125 disk.sda1.in‚0 disk.dm-1.in‚0.015625 cpu‚25.483648785996177 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2884.27734375" disk.nvme0c0n1.in‚ 9001.6796875( disk.nvme0c0n1.out‚53999.47021484375 disk.sda2.in‚0# disk.dm-0.out‚53999.47021484375Å‹âÊ”‚(:‰ ñ¿êÒø÷Ô­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚348.568' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.7136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…ëÖ/‚‚(:¤ ñ¿êÒÛÍή disk.dm-1.out‚ 271.7109375' disk.nvme0c0n1.out‚55735.5966796875# proc.memory.rssMB‚ 2884.27734375 proc.cpu.threads‚17% disk.nvme0n1.out‚55735.5966796875 disk.sda2.in‚0 disk.rbd2.out‚24.09423828125 disk.dm-0.in‚ 9002.51171875 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.59510882654994 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 271.7109375 disk.drbd2165.in‚0 disk.dm-1.in‚0.015625 disk.sda3.out‚ 271.7109375# memory_percent‚88.1166934967041# disk./.usageGB‚589.995735168457 disk.sda.in‚0.015625) proc.memory.percent‚17.60423183441162 network.sent‚ 4.852685e+06 disk.rbd1.in‚ 276.12890625 disk.sda3.in‚0.015625! disk.nvme0n1.in‚ 9002.51171875" disk.rbd1.out‚17751.1474609375 disk.rbd0.out‚0" disk.dm-0.out‚55735.5966796875# disk.nvme0c0n1.in‚ 9002.51171875( proc.memory.availableMB‚ 1946.9609375 cpu‚25.483364908912435 network.recv‚ 1.113094e+06 disk.rbd2.in‚0.015625: 4”ƒ‚(:‰ €ÀêÒþ¿Î­ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚288.582' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7164 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÚIÿ,„‚(:¡ €ÀêÒ–ÀÔ¯ disk.drbd2165.out‚0 disk.dm-1.out‚ 272.1015625" disk.rbd1.out‚17751.1474609375' disk.nvme0c0n1.out‚57494.5576171875$ disk./.usageGB‚589.9197158813477 disk.sda1.in‚0 disk.sda3.out‚ 272.1015625) proc.memory.percent‚17.60423183441162 disk.drbd2165.in‚0 disk.sda.in‚0.015625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2884.27734375 cpu‚25.483009271363116# disk.nvme0c0n1.in‚ 9002.57421875 disk.sda3.in‚0.015625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 24.185546875" disk.dm-0.out‚57494.5576171875 network.sent‚ 4.857213e+06( disk./.usagePercent‚39.5900071084076 disk.sda1.out‚0% disk.nvme0n1.out‚57494.5576171875 network.recv‚ 1.116142e+06 disk.sda.out‚ 272.1015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 286.7734375# memory_percent‚88.1166934967041( proc.memory.availableMB‚ 1946.9609375 disk.dm-1.in‚0.015625! disk.nvme0n1.in‚ 9002.57421875 disk.dm-0.in‚ 9002.57421875q`.g”…‚(:‰ ÀêÒõų¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚268.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FmCÐ1†‚(:¦ ÀêÒ¹±â¯ network.sent‚ 4.862305e+06 network.recv‚ 1.119881e+06 disk.drbd2165.out‚0% disk.nvme0n1.out‚57826.4794921875 disk.sda2.out‚0 disk.rbd1.in‚ 293.46484375# proc.memory.rssMB‚ 2884.27734375 disk.rbd2.in‚0.015625 disk.sda3.out‚ 273.09765625# disk.rbd1.out‚17751.51513671875" disk.dm-0.out‚57826.4794921875 disk.dm-1.in‚0.015625 disk.sda3.in‚0.015625 disk.sda.out‚ 273.09765625 disk.rbd2.out‚ 24.416015625 disk.nvme0n1.in‚ 9078.9609375 disk.rbd0.out‚0) proc.memory.availableMB‚ 1946.70703125) proc.memory.percent‚17.60423183441162 disk.sda1.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚57826.4794921875 disk.sda.in‚0.015625 disk.rbd0.in‚0 cpu‚25.482668153018682 proc.cpu.threads‚17) disk./.usagePercent‚39.45988129483614 disk.dm-0.in‚ 9078.9609375 disk.dm-1.out‚ 273.09765625 disk.sda2.in‚0$ memory_percent‚88.11824321746826$ disk./.usageGB‚587.9807472229004 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 9078.9609375:M¶k”‡‚(:‰ žÀêÒ¨¾¬ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚48.0 gpu.0.powerWatts‚344.908' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î|U94ˆ‚(:© žÀêÒÞñþ­$ disk./.usageGB‚587.9811325073242 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚88.11676502227783) proc.memory.availableMB‚ 1946.94921875 network.sent‚ 4.867357e+06 disk.sda1.in‚0 disk.sda3.in‚0.015625 disk.sda.out‚ 274.9453125 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 9079.79296875 network.recv‚ 1.123214e+06) disk./.usagePercent‚39.45990715159709 disk.rbd1.in‚ 298.62109375& disk.nvme0n1.out‚57850.85986328125 disk.rbd0.out‚0 disk.dm-1.out‚ 274.9453125# proc.memory.rssMB‚ 2884.27734375 disk.sda2.out‚0# disk.rbd1.out‚17751.51513671875 disk.sda3.out‚ 274.9453125! disk.nvme0n1.in‚ 9079.79296875 disk.rbd2.out‚ 24.494140625# disk.dm-0.out‚57850.85986328125 disk.sda1.out‚0 disk.sda.in‚0.015625 disk.rbd0.in‚0 disk.dm-1.in‚0.015625 cpu‚25.482407292626228# disk.nvme0c0n1.in‚ 9079.79296875( disk.nvme0c0n1.out‚57850.85986328125) proc.memory.percent‚17.60423183441162 proc.cpu.threads‚17,(.,2‰- _wandbruntime‚4392  _runtime‚4392z“ÅŠ¿ _step‚214 % lr-AdamW/pg1‚0.000265258876463829 % lr-AdamW/pg2‚0.000265258876463829 trainer/global_step‚2649 % _timestamp‚1.7843241390267243e+09 _runtime‚4392.401277455ßÌõ9Ê‹Ä % _timestamp‚1.7843241390267243e+09 _runtime‚4392.401277455 _step‚214 % lr-AdamW/pg1‚0.000265258876463829 % lr-AdamW/pg2‚0.000265258876463829 trainer/global_step‚2649Ö°±…ó2Œ- _wandbruntime‚4392  _runtime‚4392'(vî'èO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14316538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚759428679593984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6796657 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚762247766160896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2489095123456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22925021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784324139027772e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205819773284352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209032599686144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213229029376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762246394597888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14316284 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚761913490728874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22915048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22542301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2489095123456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22542307 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚762247766160896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚759428654166016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚4392.402518888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209033002313728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213194492928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759757325647872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0M¹™Ãó'ŽíO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209033002313728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213194492928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759757325647872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14316538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚759428679593984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6796657 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚762247766160896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2489095123456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22925021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784324139027772e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205819773284352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209032599686144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213229029376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762246394597888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14316284 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚761913490728874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22915048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22542301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2489095123456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22542307 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚762247766160896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚759428654166016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚4392.402518888×Í|«2- _wandbruntime‚4393  _runtime‚4393àÅŸúx óM WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7522575 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2490020158976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14321780 trainer/global_step‚2649 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14321538 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209111286092800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2487104629538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22923526 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762525320363520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2491365555712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.al¶`Ðbl.freed‚45474033 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205897259692032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214394491904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2491365555712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22550507 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚760035300204544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6799200 % _timestamp‚1.7843241404151387e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205896891600896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22933513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6799205 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22923526 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22933513 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2485760029104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759706535944192 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚760035326377984 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214429021184 _step‚216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762526691933696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22550513 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0R^þ&‘øM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14321538 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209111286092800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2487104629538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22923526 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762525320363520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2491365555712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205897259692032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214394491904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2491365555712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22550507 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚760035300204544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6799200 % _timestamp‚1.7843241404151387e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205896891600896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22933513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6799205 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22923526 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22933513 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2485760029104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759706535944192 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚760035326377984 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214429021184 _step‚216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762526691933696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22550513 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7522575 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2490020158976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14321780 trainer/global_step‚2649ØëåM2’- _wandbruntime‚4393  _runtime‚4393n•5Å“ý 2loss/SRqqt1/assignment_loss‚0.5726632549513292 -loss/FBt2/detection_loss‚0.88280980506291 _step‚217 /loss/FRt2/assignment_loss‚2.020867627046583 /loss/FRt2/detection_loss‚0.6352687279864382 .loss/FBt1/detection_loss‚0.626850741400073 trainer/global_step‚2649 0loss/FBt2/assignment_loss‚0.2603805729460722 /loss/FRt1/detection_loss‚1.5702288834967049 %loss/total_loss‚9.909035939074641 epoch‚6 % _timestamp‚1.7843241405014408e+09 2loss/SRqqt2/assignment_loss‚0.6706237491630753 2loss/FBt1/assignment_loss‚0.004464634160376116 1loss/SRqqt1/detection_loss‚0.6730707934097113 1loss/SRqqt2/detection_loss‚1.0295625115565041 _runtime‚4393.875818704 0loss/FRt1/assignment_loss‚0.9622446378948641öäÁ‚”‚ 2loss/SRqqt1/assignment_loss‚0.5726632549513292 -loss/FBt2/detection_loss‚0.88280980506291 _step‚217 /loss/FRt2/assignment_loss‚2.020867627046583 /loss/FRt2/detection_loss‚0.6352687279864382 .loss/FBt1/detection_loss‚0.626850741400073 trainer/global_step‚2649 0loss/FBt2/assignment_loss‚0.2603805729460722 /loss/FRt1/detection_loss‚1.5702288834967049 %loss/total_loss‚9.909035939074641 epoch‚6 % _timestamp‚1.7843241405014408e+09 2loss/SRqqt2/assignment_loss‚0.6706237491630753 2loss/FBt1/assignment_loss‚0.004464634160376116 1loss/SRqqt1/detection_loss‚0.6730707934097113 1loss/SRqqt2/detection_loss‚1.0295625115565041 _runtime‚4393.875818704 0loss/FRt1/assignment_loss‚0.9622446378948641Ù·…¬Ÿ•‚(:” ­ÀêÒâä–­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚277.243' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.448600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ Lt-–‚(:¢ ­ÀêÒǶޮ disk.rbd2.out‚ 24.494140625 disk.sda1.in‚0) proc.memory.availableMB‚ 1946.45703125( disk.nvme0c0n1.out‚57855.31298828125# disk.dm-0.out‚57855.31298828125$ disk./.usageGB‚587.9156875610352 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda.out‚ 275.453125 network.sent‚ 4.87255e+06 disk.drbd2165.in‚0 disk.sda3.out‚ 275.453125 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.60423183441162 disk.rbd0.in‚0 disk.sda3.in‚0.015625# disk.nvme0c0n1.in‚ 9079.79296875 disk.sda.in‚0.015625 cpu‚25.482062043469682& disk.nvme0n1.out‚57855.31298828125 disk.dm-0.in‚ 9079.79296875 network.recv‚ 1.127075e+06* disk./.usagePercent‚39.455515086339666 disk.sda2.out‚0 disk.rbd1.in‚ 302.97265625 disk.rbd1.out‚ 17752.7421875 disk.dm-1.in‚0.015625 disk.dm-1.out‚ 275.453125$ memory_percent‚88.11976909637451# proc.memory.rssMB‚ 2884.27734375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 9079.79296875á aE“—‚(:ˆ ¼ÀêÒ¬íÖ« gpu.0.gpu‚22.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.7732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÂB$˜‚(:™ ¼Àêҵخ network.sent‚ 4.935845e+06$ disk./.usageGB‚587.9157409667969 disk.drbd2165.out‚0 disk.dm-1.in‚0.015625 disk.dm-1.out‚ 275.65234375) proc.memory.percent‚17.60423183441162 network.recv‚ 1.132459e+06 disk.sda.in‚0.015625* disk./.usagePercent‚39.455518670445144 disk.rbd1.in‚ 307.94140625 disk.rbd2.out‚25.21826171875 disk.nvme0c0n1.in‚9080.75( disk.nvme0c0n1.out‚57868.04736328125 disk.sda.out‚ 275.65234375 disk.sda2.in‚0) proc.memory.availableMB‚ 1946.45703125 disk.rbd1.out‚ 17752.7421875# disk.dm-0.out‚57868.04736328125 disk.sda1.out‚0 disk.sda3.in‚0.015625& disk.nvme0n1.out‚57868.04736328125# proc.memory.rssMB‚ 2884.27734375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 275.65234375 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚9080.75 disk.sda2.out‚0$ memory_percent‚88.11976909637451 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚9080.75 cpu‚25.481792017480007Öõƒ”™‚(:‰ ËÀêÒï‹Ð« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚264.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´üJdš‚(:“ ËÀêÒÇ‚® disk.sda2.out‚0 disk.dm-0.in‚ 9138.76953125 disk.sda.out‚ 276.0390625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.in‚0.015625# disk.nvme0c0n1.in‚ 9138.76953125 disk.nvme0n1.out‚ 57872.40625 disk.rbd1.in‚ 315.0390625 disk.rbd0.out‚0$ disk./.usageGB‚587.9157028198242 disk.dm-1.in‚0.015625) proc.memory.availableMB‚ 1946.44921875) disk./.usagePercent‚39.45551611036981 disk.rbd2.out‚25.21826171875 disk.dm-0.out‚ 57872.40625 disk.dm-1.out‚ 276.0390625$ memory_percent‚88.11981678009033# proc.memory.rssMB‚ 2884.27734375" disk.nvme0c0n1.out‚ 57872.40625) proc.memory.percent‚17.60423183441162 cpu‚25.48134881614469 disk.rbd1.out‚ 17754.4296875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 276.0390625! disk.nvme0n1.in‚ 9138.76953125 disk.sda.in‚0.015625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 4.941437e+06 network.recv‚ 1.136147e+06Ž9៛‚(:” ÚÀêÒ³ëý­ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚334.396' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.87920000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÇ·-œ‚(:¢ ÚÀêÒȶž® network.sent‚ 4.946519e+06$ disk./.usageGB‚587.9156684875488" disk.nvme0c0n1.in‚ 9139.6015625' disk.nvme0c0n1.out‚57884.5224609375 disk.sda2.out‚0 disk.rbd2.out‚ 25.3369140625 disk.drbd2165.in‚0 disk.dm-1.out‚ 276.33203125) proc.memory.availableMB‚ 1946.44921875# proc.memory.rssMB‚ 2884.27734375 disk.sda3.out‚ 276.33203125 disk.rbd1.in‚ 325.109375 disk.rbd0.in‚0" disk.dm-0.out‚57884.5224609375% disk.nvme0n1.out‚57884.5224609375 disk.sda.in‚0.015625 disk.sda1.out‚0 cpu‚25.481129103812577 proc.cpu.threads‚17 disk.dm-1.in‚0.015625 disk.rbd0.out‚0 disk.dm-0.in‚ 9139.6015625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda3.in‚0.015625 disk.nvme0n1.in‚ 9139.6015625 disk.sda.out‚ 276.33203125 disk.rbd1.out‚ 17754.4296875 disk.rbd2.in‚0.015625 network.recv‚ 1.139468e+06* disk./.usagePercent‚39.455513806301994 disk.sda2.in‚0$ memory_percent‚88.11981678009033) proc.memory.percent‚17.60423183441162t¦SiŸ‚(:” éÀêÒþ¼¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚288.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.657199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i‡—%*ž‚(:Ÿ éÀêÒñŸ¬® disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.in‚0.015625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 9140.55859375 disk.sda.in‚0.015625 disk.sda2.in‚0 disk.rbd2.out‚ 25.3369140625 cpu‚25.480744366274923$ disk.nvme0n1.out‚57892.205078125 disk.dm-1.out‚ 276.61328125( proc.memory.availableMB‚ 1946.6953125# proc.memory.rssMB‚ 2884.27734375 disk.sda2.out‚0 disk.rbd1.in‚ 332.78125 disk.rbd0.out‚0$ memory_percent‚88.11831474304199) proc.memory.percent‚17.60423183441162 network.recv‚ 1.143015e+06 disk.sda3.out‚ 276.61328125 disk.sda.out‚ 276.61328125 disk.rbd1.out‚ 17755.9453125 disk.sda1.out‚0 disk.dm-0.in‚ 9140.55859375$ disk./.usageGB‚587.9157257080078 disk.sda3.in‚0.015625 disk.rbd0.in‚0! disk.dm-0.out‚57892.205078125 disk.sda1.in‚0 network.sent‚ 4.951026e+06) disk./.usagePercent‚39.45551764641501& disk.nvme0c0n1.out‚57892.205078125! disk.nvme0n1.in‚ 9140.55859375 disk.drbd2165.out‚0m~ ”Ÿ‚(:‰ øÀêÒô®î« gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚290.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦`ºŒ* ‚(:Ÿ øÀêÒÜäð® proc.cpu.threads‚17# disk./.usageGB‚588.710033416748 disk.sda.out‚ 276.96875) proc.memory.availableMB‚ 1946.69140625 disk.sda2.in‚0# memory_percent‚88.1183385848999 disk.rbd1.in‚ 343.2578125 network.sent‚ 4.956774e+06* disk./.usagePercent‚39.508824303217125 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚25.38037109375 disk.sda3.in‚0.015625! disk.nvme0n1.in‚ 9140.55859375 disk.dm-0.in‚ 9140.55859375# disk.dm-0.out‚58710.19384765625 disk.drbd2165.out‚0& disk.nvme0n1.out‚58710.19384765625 disk.sda2.out‚0 cpu‚25.480468584687625 disk.sda1.out‚0 network.recv‚ 1.146394e+06 disk.rbd0.in‚0 disk.dm-1.in‚0.015625# disk.nvme0c0n1.in‚ 9140.55859375( disk.nvme0c0n1.out‚58710.19384765625 disk.sda.in‚0.015625 disk.rbd1.out‚ 17755.9453125 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.60423183441162 disk.dm-1.out‚ 276.96875 disk.sda3.out‚ 276.96875# proc.memory.rssMB‚ 2884.27734375ÒåÝn2¡- _wandbruntime‚4470  _runtime‚4470¢y}ɢà _runtime‚4470.875632694 _step‚218 ' lr-AdamW/pg1‚0.00026486739282631375 ' lr-AdamW/pg2‚0.00026486739282631375 trainer/global_step‚2699 % _timestamp‚1.7843242175009906e+09m+'+Î£È _runtime‚4470.875632694 _step‚218 ' lr-AdamW/pg1‚0.00026486739282631375 ' lr-AdamW/pg2‚0.00026486739282631375 trainer/global_step‚2699 % _timestamp‚1.7843242175009906e+09ÚN|v±2¤- _wandbruntime‚4470  _runtime‚4470‰Qp'¥éO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273019160576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚2699 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2535411054080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚219 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23350773 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2536756443648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212966710201344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22952601 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14581863 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3272984624128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775853816750782 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚773322743074816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46293401 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22952607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6923807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23340800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6923812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚776192747034112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773657362153472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2535411054080 _runtime‚4470.876876252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚773322768502784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773657335980032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23340800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22952607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2532418276288 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209694093668352 % _timestamp‚1.7843242175021152e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648{½Ÿþô'¦îO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2535411054080 _runtime‚4470.876876252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚773322768502784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773657335980032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23340800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22952607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2532418276288 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209694093668352 % _timestamp‚1.7843242175021152e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273019160576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚2699 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2535411054080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚219 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23350773 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2536756443648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212966710201344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22952601 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14581863 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3272984624128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775853816750782 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚773322743074816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46293401 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22952607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6923807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23340800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6923812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚776192747034112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773657362153472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640Ûš/ß2§- _wandbruntime‚4472  _runtime‚4472 äLàø&¨òM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚773600650280960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 $ _timestamp‚1.784324218887906e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23359265 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7660532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23349278 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213045393167360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23359265 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536336089600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3274215711744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14586882 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2533341742646 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚773600624852992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2531997142212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _step‚220 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2537681486336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3274181182464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚776132621995204 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22960807 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209771211984896 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22960807 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚776473044369920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚776133992023606 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22960813 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚776473044369920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773935362883584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773935336710144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2537681486336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536336089600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7660778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209771580076032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820«¹ ÷©÷M E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚776471672799744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536336089600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3274215711744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14586882 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2533341742646 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚773600624852992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2531997142212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _step‚220 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2537681486336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3274181182464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚776132621995204 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22960807 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209771211984896 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22960807 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚776473044369920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚776133992023606 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22960813 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚776473044369920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773935362883584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773935336710144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2537681486336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536336089600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7660778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209771580076032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚773600650280960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 $ _timestamp‚1.784324218887906e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23359265 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7660532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23349278 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213045393167360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23359265 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7050957 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚787216857411584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚223 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787557398659072 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787216831983616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2583072374272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7050962 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790139099470336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23776525 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14847371 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7796139 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790139099470336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789794142772690 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577310789074 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213568414052352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213568045961216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784324296158367e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787557398659072 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2581726984704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14847096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3332386868224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23766552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787557372485632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362901 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7796409 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚790140471033344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787557372485632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2749tuar&¼ìO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7050957 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚787216857411584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚223 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787557398659072 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787216831983616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2583072374272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7050962 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790139099470336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23776525 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14847371 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7796139 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790139099470336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789794142772690 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577310789074 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213568414052352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213568045961216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784324296158367e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787557398659072 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2581726984704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14847096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3332386868224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23766552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787557372485632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362901 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7796409 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚790140471033344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787557372485632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2749ß6nÇ2½- _wandbruntime‚4550  _runtime‚4550AóÏOù&¾óM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23785017 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787494713761792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23371113 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14852398 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213645532368896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4550.917567023 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚787494739189760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2583997416960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2579578855754 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7799156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216979062324224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _step‚224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787835399389184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2582652020224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3333566581760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787835373215744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23775030 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23775030 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2583997416960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23371113 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7798898 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3333529955328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787835373215744|˜.¸þ&¿øM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23371107 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216979467041792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787835399389184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23785017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790419396806144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14852661 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578234255320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23371107 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7053500 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790074318045514 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚790072948017112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2582652020224 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2579578855754 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7799156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216979062324224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _step‚224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787835399389184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2582652020224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3333566581760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787835373215744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23775030 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23775030 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2583997416960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23371113 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7798898 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3333529955328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787835373215744 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23785017 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787494713761792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23371113 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14852398 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213645532368896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4550.917567023 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚787494739189760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2583997416960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BãØÒ2Ð- _wandbruntime‚4628  _runtime‚4628ºG_7œ ÑèO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23773201 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24202277 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217442734436352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚2799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220834396177920 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804085451906560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _runtim‡@ÐRe‚4628.229486051 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23773207 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804085451906560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24192304 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7178112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24192304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23773207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7934416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2628042915328 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚801457408991232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚801457435164672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚801457408991232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2628042915328 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15112523 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2629388304896 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801110946320384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚803735838822888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3392064369152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801110920892416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2623547902182 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217442366345216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804085451906560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _runtime‚4628.229486051 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23773207 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804085451906560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24192304 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7178112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24192304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23773207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7934416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2628042915328 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚801457408991232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚801457435164672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚801457408991232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2628042915328 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15112523 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2629388304896 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801110946320384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚803735838822888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175㚟!Š2Ó- _wandbruntime‚4629  _runtime‚4629¢¶Øù&ÔóM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15117812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4629.613787877 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217520220844032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220913481321984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3393260477952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628967950848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24200782 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2630313347584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚804365749242368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801388802670592 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15118052 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3393225948672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843243762389011e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7180655 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7937162 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚2799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23781407 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7937397 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2624471368428 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23781407 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚801735435894784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚801735409721344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801388828098560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚801735435894784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217519852752896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2625815968862 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7180650 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926Ï mÕøM PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801388802670592 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15118052 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3393225948672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843243762389011e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7180655 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7937162 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚2799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23781407 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7937397 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2624471368428 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segE”œà‘ment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23781407 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚801735435894784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚801735409721344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801388828098560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚801735435894784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217519852752896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2625815968862 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7180650 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15117812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4629.613787877 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217520220844032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220913481321984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3393260477952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628967950848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24200782 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2630313347584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚804365749242368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2äŠU2Ö- _wandbruntime‚4629  _runtime‚4629¤´¥æ× %loss/total_loss‚9.368680574163317 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5125635036830907 2loss/SRqqt2/assignment_loss‚0.7065877213254343 2loss/FBt2/assignment_loss‚0.023565920306106047 /loss/FRt2/detection_loss‚0.6523365062941873 1loss/SRqqt1/detection_loss‚0.6302548455088738 trainer/global_step‚2799 $ _timestamp‚1.784324376325252e+09 _runtime‚4629.699616041 0loss/FRt1/assignment_loss‚0.7974674684660775 /loss/FRt2/assignment_loss‚2.020068204320629 /loss/FRt1/detection_loss‚1.6156000621627928 /loss/FBt1/detection_loss‚0.5723167529569999 /loss/FBt2/detection_loss‚0.8037201591068879 _step‚229 2loss/FBt1/assignment_loss‚0.018948978475524316 1loss/SRqqt2/detection_loss‚1.0152504515567131ä|æq ؆ 2loss/FBt1/assignment_loss‚0.018948978475524316 1loss/SRqqt2/detection_loss‚1.0152504515567131 %loss/total_loss‚9.368680574163317 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5125635036830907 2loss/SRqqt2/assignment_loss‚0.7065877213254343 2loss/FBt2/assignment_loss‚0.023565920306106047 /loss/FRt2/detection_loss‚0.6523365062941873 1loss/SRqqt1/detection_loss‚0.6302548455088738 trainer/global_step‚2799 $ _timestamp‚1.784324376325252e+09 _runtime‚4629.699616041 0loss/FRt1/assignment_loss‚0.7974674684660775 /loss/FRt2/assignment_loss‚2.020068204320629 /loss/FRt1/detection_loss‚1.6156000621627928 /loss/FBt1/detection_loss‚0.5723167529569999 /loss/FBt2/detection_loss‚0.8037201591068879 _step‚229å|Qì’Ù‚(:‡ ÂêÒþĬ gpu.0.gpu‚4.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^6»ˆ0Ú‚(:¥ ÂêÒ¡ß»¯ disk.drbd2165.out‚0' disk.nvme0c0n1.out‚58790.5927734375 disk.rbd0.out‚0 disk.rbd2.out‚ 26.3603515625 disk.dm-0.in‚ 9310.51171875 network.recv‚ 1.186845e+06) disk./.usagePercent‚39.45736576480469 disk.rbd0.in‚0 disk.rbd2.in‚0.015625$ disk./.usageGB‚587.9432640075684 disk.sda1.out‚0 disk.sda3.in‚0.015625 disk.sda.in‚0.015625 disk.sda.out‚ 282.06640625" disk.dm-0.out‚58790.5927734375% disk.nvme0n1.out‚58790.5927734375 disk.sda2.out‚0) proc.memory.percent‚17.60423183441162 cpu‚25.48002701183751 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0.015625! disk.nvme0n1.in‚ 9310.51171875# proc.memory.rssMB‚ 2884.27734375 network.sent‚ 5.128229e+06 disk.sda2.in‚0 disk.rbd1.out‚ 17761.6953125 disk.dm-1.out‚ 282.06640625# disk.nvme0c0n1.in‚ 9310.51171875$ memory_percent‚88.12415599822998 disk.sda3.out‚ 282.06640625 disk.rbd1.in‚ 443.26953125) proc.memory.availableMB‚ 1945.73828125ã´cÕžÛ‚(:“ ¬ÂêÒ†ðû­ gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚339.239' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.84779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©16X!Ü‚(:– ¬ÂêÒð¿‹° disk.rbd2.out‚26.42333984375) proc.memory.availableMB‚ 1946.05078125# proc.memory.rssMB‚ 2884.27734375 cpu‚25.48821258981905 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 9311.46875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 5.191173e+06 disk.nvme0n1.in‚ 9311.46875 disk.dm-0.in‚ 9311.46875" disk.dm-0.out‚58797.9892578125 disk.drbd2165.out‚0$ memory_percent‚88.12224864959717 network.recv‚ 1.191466e+06) disk./.usagePercent‚39.45735194039784 disk.sda3.in‚0.015625 disk.sda3.out‚ 282.2578125 disk.drbd2165.in‚0 disk.dm-1.in‚0.015625 disk.rbd2.in‚0.015625 proc.cpu.threads‚17 disk.sda2.in‚0' disk.nvme0c0n1.out‚58797.9892578125 disk.sda.out‚ 282.2578125 disk.rbd1.in‚ 451.21875 disk.rbd1.out‚ 17761.6953125 disk.dm-1.out‚ 282.2578125) proc.memory.percent‚17.60423183441162# disk./.usageGB‚587.943058013916% disk.nvme0n1.out‚58797.9892578125 disk.sda.in‚0.0156253}ÒB2Ý- _wandbruntime‚4655  _runtime‚46555ƒèG®=Þ¨{ 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6398079991340637 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5783505439758301 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5519594550132751 jet/accuracy_5_of_5‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8555814623832703 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.694543719291687 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4603269696235657 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12438717484474182 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14379215240478516 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941217303276062 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1379636973142624 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1984691321849823 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7437645196914673 +jet/accuracy_3_of_3‚0.06341839581727982 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6421576738357544 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06333683431148529 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6250848770141602 +particle/specificity‚0.9100809693336487 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12259030342102051 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5962514877319336 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7895880937576294 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 *jet/accuracy_2_of_4‚0.7010251879692078 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6002215147018433 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9846512675285339 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.537815511226654 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9941798448562622 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991936445236206 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6884897351264954 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8557813763618469 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10518687963485718 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6419540047645569 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.12740503251552582 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 *jet/accuracy_1_of_4‚0.9566853046417236 /particle/accuracy_4_of_4‚0.9454245567321777 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 epoch‚6 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9162259697914124 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06279687583446503 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9852662086486816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6058520674705505 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027880338951945305 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015034188516438007 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6971562504768372 +jet/accuracy_2_of_3‚0.48406633734703064 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.995418906211853 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11550327390432358 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_3‚0.8989847302436829 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21599163115024567 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6282747983932495 particle/accuracy_3_of_3‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_5‚1 _step‚230 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01764669083058834 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25954562425613403 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_5‚1 5 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46974977850914 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987532913684845 'particle/f_score‚0.6237397193908691 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6428002715110779 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6839758157730103 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09398383647203445 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5999327898025513 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6804394721984863 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9943991899490356 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7180338501930237 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7544360160827637 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919853210449219 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6062850952148438 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.129764586687088 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.715155303478241 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7508125305175781 +jet/accuracy_3_of_4‚0.07907283306121826 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3219817578792572 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14908906817436218 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06021692231297493 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6758744120597839 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12578842043876648 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9954027533531189 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0664077177643776 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.674422025680542 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7validation_average_jet_accuracy‚0.32508382201194763 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3503921926021576 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16025331616401672 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13204963505268097 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9886156916618347 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9949836730957031 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9866213202476501 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1158333271741867 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06115129962563515 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4975000023841858 *jet/accuracy_1_of_2‚0.7502734065055847 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 )loss/val_total_loss‚9.170467638084506 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7944632768630981 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6442486643791199 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028606252744793892 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1689203679561615 trainer/global_step‚2799 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13989093899726868 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7309687733650208 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8909159898757935 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6218851804733276 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46696382761001587 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11739659309387207 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3476909101009369 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 % _timestamp‚1.7843244023456507e+09 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14057058095932007 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0664077177643776 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5710955858230591 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7002277374267578 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7212138175964355 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6482732892036438 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11965509504079819 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937227368354797 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6992607116699219 _runtime‚4655.720910911 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2567308247089386 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6131991147994995 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01507073175162077 *jet/accuracy_1_of_1‚0.2767612934112549 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7544360160827637 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923118352890015 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6946661472320557 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1385807991027832 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11449803411960602 (particle/accuracy‚0.8439708948135376 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863381385803223 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5999327898025513 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5951364040374756 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007978411391377449 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.6022452116012573 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8023054003715515 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5823015570640564 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1504906862974167 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5416224002838135 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4183062016963959 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5958897471427917 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12438717484474182 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09759601205587387 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5963541269302368 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944577217102051 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 jet/accuracy_3_of_5‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9914049506187439 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928325414657593 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14105618000030518 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6423929929733276 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.31282469630241394 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1331223100423813 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.792101263999939 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1059766635298729 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872ŒNœf|ß­{ 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6398079991340637 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5783505439758301 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5519594550132751 jet/accuracy_5_of_5‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8555814623832703 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.694543719291687 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4603269696235657 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12438717484474182 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14379215240478516 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941217303276062 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1379636973142624 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1984691321849823 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7437645196914673 +jet/accuracy_3_of_3‚0.06341839581727982 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6421576738357544 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06333683431148529 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6250848770141602 +particle/specificity‚0.9100809693336487 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12259030342102051 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5962514877319336 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7895880937576294 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 *jet/accuracy_2_of_4‚0.7010251879692078 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6002215147018433 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9846512675285339 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.537815511226654 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9941798448562622 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991936445236206 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6884897351264954 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8557813763618469 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10518687963485718 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6419540047645569 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.12740503251552582 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 *jet/accuracy_1_of_4‚0.9566853046417236 /particle/accuracy_4_of_4‚0.9454245567321777 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 epoch‚6 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9162259697914124 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06279687583446503 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9852662086486816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6058520674705505 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027880338951945305 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015034188516438007 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6971562504768372 +jet/accuracy_2_of_3‚0.48406633734703064 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.995418906211853 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11550327390432358 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_3‚0.8989847302436829 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21599163115024567 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6282747983932495 particle/accuracy_3_of_3‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_5‚1 _step‚230 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01764669083058834 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25954562425613403 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_5‚1 5 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46974977850914 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987532913684845 'particle/f_score‚0.6237397193908691 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6428002715110779 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6839758157730103 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09398383647203445 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5999327898025513 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6804394721984863 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9943991899490356 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7180338501930237 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7544360160827637 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919853210449219 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6062850952148438 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.129764586687088 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.715155303478241 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7508125305175781 +jet/accuracy_3_of_4‚0.07907283306121826 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3219817578792572 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.997023|n£T7!8208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14908906817436218 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06021692231297493 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6758744120597839 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12578842043876648 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9954027533531189 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0664077177643776 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.674422025680542 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7validation_average_jet_accuracy‚0.32508382201194763 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3503921926021576 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16025331616401672 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13204963505268097 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9886156916618347 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9949836730957031 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9866213202476501 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1158333271741867 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06115129962563515 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4975000023841858 *jet/accuracy_1_of_2‚0.7502734065055847 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 )loss/val_total_loss‚9.170467638084506 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7944632768630981 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6442486643791199 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028606252744793892 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1689203679561615 trainer/global_step‚2799 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13989093899726868 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7309687733650208 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8909159898757935 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6218851804733276 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46696382761001587 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11739659309387207 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3476909101009369 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 % _timestamp‚1.7843244023456507e+09 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14057058095932007 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0664077177643776 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5710955858230591 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7002277374267578 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7212138175964355 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6482732892036438 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11965509504079819 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937227368354797 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6992607116699219 _runtime‚4655.720910911 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2567308247089386 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6131991147994995 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01507073175162077 *jet/accuracy_1_of_1‚0.2767612934112549 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7544360160827637 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923118352890015 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6946661472320557 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1385807991027832 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11449803411960602 (particle/accuracy‚0.8439708948135376 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863381385803223 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5999327898025513 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5951364040374756 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007978411391377449 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.6022452116012573 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8023054003715515 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5823015570640564 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1504906862974167 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5416224002838135 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4183062016963959 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5958897471427917 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12438717484474182 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09759601205587387 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5963541269302368 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944577217102051 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 jet/accuracy_3_of_5‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9914049506187439 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928325414657593 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14105618000030518 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6423929929733276 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.31282469630241394 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1331223100423813 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.792101263999939 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1059766635298729 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872æ¡Ð`…&à‚(:› »ÂêÒß°µ® disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 284.359375 disk.sda.out‚ 284.359375 disk.rbd2.out‚26.42333984375 proc.memory.rssMB‚2886.5# disk./.usageGB‚587.947868347168 disk.sda1.out‚0 disk.dm-1.out‚ 284.359375 disk.rbd0.in‚0( proc.memory.percent‚17.6177978515625 cpu‚25.48810946321202 disk.dm-0.in‚ 9418.25390625 disk.rbd0.out‚0! disk.nvme0n1.in‚ 9418.25390625 disk.sda2.in‚0 disk.rbd1.out‚ 17762.8203125 disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.0703125( disk.nvme0c0n1.out‚60756.48193359375& disk.nvme0n1.out‚60756.48193359375 disk.sda.in‚ 0.0703125 disk.sda2.out‚0 disk.rbd1.in‚ 477.33984375( proc.memory.availableMB‚ 2915.6328125) disk./.usagePercent‚39.45767476589852# disk.nvme0c0n1.in‚ 9418.25390625# disk.dm-0.out‚60756.48193359375$ memory_percent‚82.20438957214355 proc.cpu.threads‚17 network.sent‚ 5.195697e+06 network.recv‚ 1.19486e+06 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0703125ØF3”á‚(:‰ »ÂêÒŒŠ±® gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.912' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øHÔâ‚(:‰ ÊÂêÒ® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚344.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RK]*ã‚(:Ÿ ÊÂêÒéÀ€±* disk./.usagePercent‚39.459573829787516 disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 9418.25390625 disk.sda3.in‚ 0.0703125 disk.rbd1.in‚ 481.5859375 disk.drbd2165.out‚0 disk.dm-1.out‚ 286.48046875) proc.memory.availableMB‚ 2915.86328125 disk.rbd1.out‚ 17762.8203125 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125$ memory_percent‚82.20298290252686! disk.nvme0n1.in‚ 9418.25390625% disk.nvme0n1.out‚60788.9208984375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚60788.9208984375 network.recv‚ 1.19989e+06 disk.sda.in‚ 0.0703125 disk.sda.out‚ 286.48046875 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚26.42333984375' disk.nvme0c0n1.out‚60788.9208984375 network.sent‚ 5.25335e+06$ disk./.usageGB‚587.9761657714844 disk.dm-0.in‚ 9418.25390625 disk.drbd2165.in‚0( proc.memory.percent‚17.6177978515625 disk.sda3.out‚ 286.48046875 proc.memory.rssMB‚2886.5 cpu‚25.48785223274706 proc.cpu.threads‚17=ôLžä‚(:“ ÙÂêÒâü ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚275.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.17759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ £U#å‚(:˜ ÙÂêÒ÷–Ï®" disk.nvme0c0n1.in‚ 9419.2109375 disk.rbd0.out‚0( proc.memory.percent‚17.6177978515625' disk.nvme0c0n1.out‚60807.0087890625 disk.drbd2165.out‚0% disk.nvme0n1.out‚60807.0087890625 disk.sda2.in‚0 disk.rbd1.out‚17763.73828125) proc.memory.availableMB‚ 2915.52734375 network.recv‚ 1.20349e+06 disk.rbd1.in‚ 496.9296875 disk.rbd2.out‚26.42333984375 disk.sda1.in‚0$ memory_percent‚82.20503330230713 disk.sda.out‚ 287.859375 disk.rbd0.in‚0" disk.dm-0.out‚60807.0087890625 network.sent‚ 5.258391e+06$ disk./.usageGB‚587.9747276306152 disk.sda3.out‚ 287.859375 disk.sda2.out‚0 disk.dm-0.in‚ 9419.2109375 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125 disk.nvme0n1.in‚ 9419.2109375 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 proc.memory.rssMB‚2886.5) disk./.usagePercent‚39.45947731494711 disk.sda.in‚ 0.0703125 disk.dm-1.out‚ 287.859375 cpu‚25.487441013195237 proc.cpu.threads‚17Ù=Œbžæ‚(:“ èÂêÒçæÏ« gpu.0.gpu‚94.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚329.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.89299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ôëåç‚(:“ èÂêÒÌ®á­# memory_percent‚82.2066068649292 network.recv‚ 1.206792e+06 disk.dm-0.out‚ 60812.3515625 disk.sda1.in‚0 disk.dm-1.in‚ 0.0703125" disk.nvme0c0n1.in‚ 9419.2109375 disk.dm-1.out‚ 289.08203125 disk.sda3.out‚ 289.08203125 disk.sda.in‚ 0.0703125 disk.sda2.in‚0 disk.rbd1.in‚ 506.015625 disk.rbd1.out‚ 17763.8359375( proc.memory.percent‚17.6177978515625 disk.drbd2165.out‚0 network.sent‚ 5.263478e+06 disk.nvme0n1.in‚ 9419.2109375" disk.nvme0n1.out‚ 60812.3515625 disk.sda.out‚ 289.08203125 disk.rbd2.out‚26.48583984375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.0703125 proc.cpu.threads‚17* disk./.usagePercent‚39.459392064438205 disk.rbd2.in‚0.015625) proc.memory.availableMB‚ 2915.26953125 proc.memory.rssMB‚2886.5 cpu‚25.48717362192386$ disk./.usageGB‚587.9734573364258 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 9419.2109375$ disk.nvme0c0n1.out‚ 60812.3515625 disk.sda2.out‚0ýS$”è‚(:‰ ÷ÂêÒŸË¿¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äú½)é‚(:ž ÷ÂêÒ¿ÖŠ®' disk.nvme0c0n1.out‚60828.1904296875 disk.sda.out‚ 290.53515625" disk.dm-0.out‚60828.1904296875 disk.drbd2165.in‚0 network.sent‚ 5.269209e+06 disk.dm-0.in‚ 9517.98828125 disk.dm-1.out‚ 290.53515625( proc.memory.availableMB‚ 2915.2734375) disk./.usagePercent‚39.45937132782793 disk.sda3.out‚ 290.53515625 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚82.20658302307129! disk.nvme0n1.in‚ 9517.98828125 disk.sda.in‚ 0.0703125 disk.rbd1.in‚ 506.015625 disk.rbd1.out‚ 17764.3671875 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 cpu‚25.48678944240701 network.recv‚ 1.210364e+06 disk.sda2.out‚0 disk.rbd2.out‚26.48583984375 disk.sda1.out‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 9517.98828125% disk.nvme0n1.out‚60828.1904296875 disk.drbd2165.out‚0( proc.memory.percent‚17.6177978515625$ disk./.usageGB‚587.9731483459473 disk.sda3.in‚ 0.0703125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.0703125 proc.memory.rssMB‚2886.5©dT2ê- _wandbruntime‚4738  _runtime‚4738¤-Y_Éëà ' lr-AdamW/pg1‚0.00026364378181980316 ' lr-AdamW/pg2‚0.00026364378181980316 trainer/global_step‚2849 % _timestamp‚1.7843244846579864e+09 _runtime‚4738.032682477 _step‚231œçuÙÎìÈ ' lr-AdamW/pg1‚0.00026364378181980316 ' lr-AdamW/pg2‚0.00026364378181980316 trainer/global_step‚2849 % _timestamp‚1.7843244846579864e+09 _runtime‚4738.032682477 _step‚231çRÉN2í- _wandbruntime‚4738  _runtime‚4738Ãä“×í'îçO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992550 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2681546849792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8106446 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2675614146732 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221364798976000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚821865628391936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819185453105152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24315513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24677043 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24315513 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 4738.03395018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7314175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8106199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221365167067136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24687016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680201460224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676958747054 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819185426931712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224824025559552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚821866999954944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24315507 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3459226583552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819185453105152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15420374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224824426089984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784324484659185e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚821866999954944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821865628391936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821507902908846 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24687016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821506532880556 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7314180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819185426931712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224824025559552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚821866999954944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24315507 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3459226583552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819185453105152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15420374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224824426089984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784324484659185e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚821866999954944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821865628391936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821507902908846 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24687016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821506532880556 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7314180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819185426931712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992550 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2681546849792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8106446 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2675614146732 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aMDeviceStatsMonitor.on_traiFßò:n_batch_start/inactive_split_bytes.large_pool.freed‚221364798976000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚821865628391936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819185453105152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24315513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24677043 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24315513 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 4738.03395018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7314175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8106199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221365167067136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24687016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680201460224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676958747054 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819185426931712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973è}êº2ð- _wandbruntime‚4739  _runtime‚4739bÎÿø&ñòM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819463453835264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 _step‚233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819463453835264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24323719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821786708153380 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821785338124978 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24323713 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2681126495744 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224903108983296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822145925727744 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224902708460032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24323713 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822145925727744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8109204 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 $ _timestamp‚1.784324486045803e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4739.420628907 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221442285383680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2849 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819463427661824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460423076352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460455508480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819463427661824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7316723 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24695508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2676537612978 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24323719 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15425927 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15425682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819108825939968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24685521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819108800512000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822144554157568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682471892480 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24685521¤PN”ý&ò÷M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24695508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7316718 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2681126495744 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677882213412 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822145925727744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8109204 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 $ _timestamp‚1.784324486045803e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4739.420628907 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221442285383680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2849 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819463427661824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460423076352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460455508480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819463427661824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7316723 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24695508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2676537612978 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24323719 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15425927 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15425682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819108825939968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24685521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819108800512000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822144554157568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682471892480 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24685521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819463453835264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 _step‚233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819463453835264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24323719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821786708153380 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821785338124978 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24323713 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2681126495744 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224903108983296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822145925727744 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224902708460032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24323713 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚835813352391168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7441330 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832725033070592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15685808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚833085463437312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚833085489610752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚833085463437312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228758143489536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25102795 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25102795 % _timestamp‚1.7843245636086583e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8244483 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228758546117120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835446858902464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24725807 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2721851259840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828602 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25112768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835448228930754 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832725007642624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚835813352391168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727862780416 trainer/global_step‚2899 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚835811980828160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225239487451136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15685808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚833085463437312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚833085489610752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚833085463437312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228758143489536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25102795 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25102795 % _timestamp‚1.7843245636086583e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8244483 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228758546117120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835446858902464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24725807 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2721851259840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828602 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25112768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835448228930754 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832725007642624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚835813352391168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727862780416 trainer/global_step‚2899 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚835811980828160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225239487451136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚835813352391168ìEBH2ˆ- _wandbruntime‚4818  _runtime‚4818»¸«b#&‰óM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚833363490340864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25111273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228836818271744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7443873 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2727442426368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24734019 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15691065 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚833363464167424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49845286 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2727442426368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24734013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836090906593792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚833002889420800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25111273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚833363490340864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225316605767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚835727034175288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2722774726086 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728787823104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855279 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728787823104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855279 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25111273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228836818271744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7443873 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2727442426368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24734019 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15691065 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚833363464167424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49845286 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2727442426368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24734013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836090906593792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5íh2Û2‹- _wandbruntime‚4818  _runtime‚4818D%)Œ€ 1loss/SRqqt1/detection_loss‚0.6027632930404786 /loss/FBt2/detection_loss‚0.7224004174824115 trainer/global_step‚2899 2loss/SRqqt2/assignment_loss‚0.6823412534454993 2loss/FBt1/assignment_loss‚0.030410096094992203 _runtime‚ 4818.45307063 1loss/SRqqt2/detection_loss‚1.0327090220400852 epoch‚7 _step‚238 0loss/FRt2/assignment_loss‚1.9791052701546974 2loss/SRqqt1/assignment_loss‚0.5003045760506861 /loss/FRt1/detection_loss‚1.3835225837522769 /loss/FRt2/detection_loss‚0.6434585241345088 /loss/FBt1/detection_loss‚0.4935209790617659 %loss/total_loss‚8.904917394330983 % _timestamp‚1.7843245650787425e+09 /loss/FRt1/assignment_loss‚0.767640128666657 1loss/FBt2/assignment_loss‚0.06674125040692364¹Ü»X … 1loss/SRqqt2/detection_loss‚1.0327090220400852 epoch‚7 _step‚238 0loss/FRt2/assignment_loss‚1.9791052701546974 2loss/SRqqt1/assignment_loss‚0.5003045760506861 /loss/FRt1/detection_loss‚1.3835225837522769 /loss/FRt2/detection_loss‚0.6434585241345088 /loss/FBt1/detection_loss‚0.4935209790617659 %loss/total_loss‚8.904917394330983 % _timestamp‚1.7843245650787425e+09 /loss/FRt1/assignment_loss‚0.767640128666657 1loss/FBt2/assignment_loss‚0.06674125040692364 1loss/SRqqt1/detection_loss‚0.6027632930404786 /loss/FBt2/detection_loss‚0.7224004174824115 trainer/global_step‚2899 2loss/SRqqt2/assignment_loss‚0.6823412534454993 2loss/FBt1/assignment_loss‚0.030410096094992203 _runtime‚ 4818.45307063îóåùl”Ž‚(:‰ àÃêÒ‹íÁ« gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚330.385' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.077 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l÷§%‚(:š àÃêÒîÆÍ­" disk.rbd1.out‚17764.9853515625' disk.nvme0c0n1.out‚60889.2763671875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 294.6171875' proc.memory.availableMB‚ 1943.078125 proc.memory.rssMB‚2886.5( proc.memory.percent‚17.6177978515625) disk./.usagePercent‚39.45970925777312 disk.rbd0.in‚0 disk.dm-0.in‚ 9606.94921875" disk.dm-0.out‚60889.2763671875 disk.sda3.out‚ 294.6171875# memory_percent‚88.1403923034668 network.sent‚ 5.362393e+06 disk.rbd1.in‚506.3125 disk.rbd2.out‚ 28.6376953125# disk.nvme0c0n1.in‚ 9606.94921875% disk.nvme0n1.out‚60889.2763671875 disk.sda.in‚ 0.0703125 cpu‚25.484763107270272 proc.cpu.threads‚17 network.recv‚ 1.235893e+06$ disk./.usageGB‚587.9781837463379 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 294.6171875 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.sda3.in‚ 0.0703125 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 9606.94921875ÞÓîåž‚(:“ ïÃêÒ£¬±¬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%?Œ,/‘‚(:¤ ïÃêÒ„œ° disk.rbd0.out‚0# disk.dm-0.out‚60895.60009765625 disk.sda1.out‚0 disk.sda3.out‚ 294.81640625 proc.memory.rssMB‚2886.5 disk.drbd2165.out‚0 disk.rbd1.in‚ 506.3515625 disk.rbd2.in‚0.015625 disk.sda1.in‚0( proc.memory.percent‚17.6177978515625" disk.rbd1.out‚17765.0947265625 disk.rbd2.out‚ 28.6376953125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 9606.94921875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 9606.94921875( proc.memory.availableMB‚ 1943.0390625 cpu‚25.484422190112856) disk./.usagePercent‚39.45970618568271 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 294.81640625( disk.nvme0c0n1.out‚60895.60009765625& disk.nvme0n1.out‚60895.60009765625 proc.cpu.threads‚17 network.sent‚ 5.425966e+06 network.recv‚ 1.241037e+06 disk.sda.in‚ 0.0703125 disk.sda3.in‚ 0.0703125! disk.nvme0n1.in‚ 9606.94921875$ disk./.usageGB‚587.9781379699707 disk.sda.out‚ 294.81640625 disk.sda2.in‚0# memory_percent‚88.1406307220459g–œ ’‚(:’ þÃêҌջ® gpu.0.gpu‚25.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÀE1/“‚(:¤ þÃêÒŨ£° disk.rbd1.in‚ 506.3515625" disk.rbd1.out‚17765.0947265625 disk.dm-1.out‚ 295.37109375( proc.memory.percent‚17.6177978515625 network.recv‚ 1.244369e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚2886.5 disk.drbd2165.in‚0 disk.sda3.out‚ 295.37109375( disk.nvme0c0n1.out‚60903.29345703125) proc.memory.availableMB‚ 1943.03515625 proc.cpu.threads‚17* disk./.usagePercent‚39.459708233742994 disk.dm-0.in‚ 9689.86328125 disk.drbd2165.out‚0$ memory_percent‚88.14065456390381 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125 disk.sda.out‚ 295.37109375# disk.nvme0c0n1.in‚ 9689.86328125! disk.nvme0n1.in‚ 9689.86328125& disk.nvme0n1.out‚60903.29345703125 cpu‚25.4841623912346 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 28.783203125# disk.dm-0.out‚60903.29345703125 disk.sda.in‚ 0.0703125 network.sent‚ 5.430484e+06$ disk./.usageGB‚587.9781684875488 disk.sda1.out‚0…Þ+Ÿ”‚(:” ÄêÒåëú« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.444799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºíšg•‚(:“ ÄêÒŠ´ð®! disk.dm-0.out‚60918.169921875 cpu‚25.483868789392886 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 9690.37890625$ memory_percent‚88.14065456390381 disk.dm-1.in‚ 0.0703125) proc.memory.availableMB‚ 1943.03515625 proc.cpu.threads‚17 network.sent‚ 5.43559e+06* disk./.usagePercent‚39.459926352162164 disk.sda3.out‚297.0625 disk.drbd2165.out‚0( proc.memory.percent‚17.6177978515625 disk.sda1.in‚0 disk.dm-1.out‚297.0625# disk.nvme0c0n1.in‚ 9690.37890625 disk.rbd0.in‚0 disk.drbd2165.in‚0 network.recv‚ 1.247967e+06 disk.sda3.in‚ 0.0703125& disk.nvme0c0n1.out‚60918.169921875 disk.sda.in‚ 0.0703125 disk.sda.out‚297.0625 disk.sda2.out‚0 disk.rbd1.in‚ 506.3515625 disk.rbd0.out‚0$ disk./.usageGB‚587.9814186096191! disk.nvme0n1.in‚ 9690.37890625$ disk.nvme0n1.out‚60918.169921875 disk.rbd2.out‚ 28.783203125 disk.sda1.out‚0 proc.memory.rssMB‚2886.5 disk.sda2.in‚0" disk.rbd1.out‚17765.2353515625î”Ï–‚(:’ œÄêÒ²³û« gpu.0.gpu‚83.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0püýs'—‚(:œ œÄêÒøÞÌ® disk.rbd1.in‚506.4375 disk.dm-1.in‚ 0.0703125 cpu‚25.48356493727025$ disk./.usageGB‚587.9888763427734" disk.rbd1.out‚17765.2353515625 disk.rbd2.out‚29.08154296875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0703125# disk.nvme0c0n1.in‚ 9690.86328125& disk.nvme0n1.out‚60949.90576171875( disk./.usagePercent‚39.4604268468916 disk.sda2.in‚0& proc.memory.availableMB‚ 1942.78125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 299.9453125 network.sent‚ 5.441238e+06 network.recv‚ 1.251501e+06# disk.dm-0.out‚60949.90576171875 disk.sda1.in‚0( disk.nvme0c0n1.out‚60949.90576171875! disk.nvme0n1.in‚ 9690.86328125$ memory_percent‚88.14220428466797( proc.memory.percent‚17.6177978515625 disk.sda1.out‚0 disk.sda.in‚ 0.0703125 disk.sda.out‚ 299.9453125 proc.memory.rssMB‚2886.5 disk.dm-0.in‚ 9690.86328125 disk.sda2.out‚0 disk.sda3.out‚ 299.9453125 proc.cpu.threads‚17|–Ã2˜- _wandbruntime‚4896  _runtime‚4896gÝ}•É™Ã ' lr-AdamW/pg1‚0.00026278733768104844 ' lr-AdamW/pg2‚0.00026278733768104844 trainer/global_step‚2949 % _timestamp‚1.7843246426751924e+09 _runtime‚4896.050378454 _step‚239‡ÝÑ+ÎšÈ ' lr-AdamW/pg1‚0.00026278733768104844 ' lr-AdamW/pg2‚0.00026278733768104844 trainer/global_step‚2949 % _timestamp‚1.7843246426751924e+09 _runtime‚4896.050378454 _step‚239ï ‰2›- _wandbruntime‚4896  _runtime‚4896ý)¾î'œèO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25528547 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚849758333264384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25136113 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25136107 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚ 4896.05085351 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229113807835136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843246426760252e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚846985526116352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25538520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25136113 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7568475 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3578354845696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849388554952662 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚846619096551424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2774178711040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232692197217280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229113439744000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578389382144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚849759704827392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8383449 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2769432973270 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2768088372948 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚849758333264384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2774178711040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25528547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25136107 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8383190 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚849758333264384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25136113 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25136107 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚ 4896.05085351 IDeviceStatsMonitor.on_train_batch_start/allocation.all.currU—3ñYent‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229113807835136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843246426760252e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚846985526116352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25538520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25136113 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7568475 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3578354845696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849388554952662 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚846619096551424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2774178711040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232692197217280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229113439744000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578389382144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚849759704827392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8383449 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2769432973270 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2768088372948 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚849758333264384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2774178711040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25528547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25136107 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8383190 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7571018 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15957180 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚846897003757568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15956930 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681338 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2769011839194 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579583737856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8385912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850038630600192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2775103753728 % _timestamp‚1.7843246440625987e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2773758356992 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850037259030016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚4897.437482684 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847263500673024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7571023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚241 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847263526846464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2773758356992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2949 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232770475360256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229191294242816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995pþ& øM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579583737856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8385912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850038630600192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2775103753728 % _timestamp‚1.7843246440625987e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2773758356992 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850037259030016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚4897.437482684 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847263500673024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7571023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚241 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847263526846464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2773758356992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2949 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232770475360256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229191294242816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25537025 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847263500673024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2770356439628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7571018 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15957180 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚846897003757568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15956930 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681338 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2769011839194 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128ñdWø2¡- _wandbruntime‚4897  _runtime‚4897G¦ýþ¢ %loss/total_loss‚10.09813542835804 epoch‚7 % _timestamp‚1.7843246441480556e+09 2loss/FBt1/assignment_loss‚0.017488598122286994 /loss/FRt2/detection_loss‚0.6110730103726396 1loss/SRqqt2/detection_loss‚1.0796168842916876 /loss/FBt2/detection_loss‚1.3092227379480998 trainer/global_step‚2949 _step‚242 /loss/FRt1/assignment_loss‚0.973999301413112 0loss/FRt2/assignment_loss‚2.0198153405131745 1loss/FBt2/assignment_loss‚0.09256415923397678 1loss/SRqqt1/detection_loss‚0.7007891680323519 _runtime‚4897.522559998 2loss/SRqqt2/assignment_loss‚0.6654992765640465 /loss/FRt1/detection_loss‚1.5029394257704372 /loss/FBt1/detection_loss‚0.5095371170730617 2loss/SRqqt1/assignment_loss‚0.6155904090231654:,}› £† /loss/FRt1/detection_loss‚1.5029394257704372 /loss/FBt1/detection_loss‚0.5095371170730617 2loss/SRqqt1/assignment_loss‚0.6155904090231654 %loss/total_loss‚10.09813542835804 epoch‚7 % _timestamp‚1.7843246441480556e+09 2loss/FBt1/assignment_loss‚0.017488598122286994 /loss/FRt2/detection_loss‚0.6110730103726396 1loss/SRqqt2/detection_loss‚1.0796168842916876 /loss/FBt2/detection_loss‚1.3092227379480998 trainer/global_step‚2949 _step‚242 /loss/FRt1/assignment_loss‚0.973999301413112 0loss/FRt2/assignment_loss‚2.0198153405131745 1loss/FBt2/assignment_loss‚0.09256415923397678 1loss/SRqqt1/detection_loss‚0.7007891680323519 _runtime‚4897.522559998 2loss/SRqqt2/assignment_loss‚0.6654992765640465ò¥?2ù”¤‚(:‰ «ÄêÒ´ÜǬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•âÅ¥‚(:‹ «ÄêÒÀ­ê®# disk.nvme0n1.out‚60970.98828125 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚29.08154296875 proc.cpu.threads‚17 cpu‚25.483206724237192 network.sent‚ 5.446286e+06 disk.dm-0.in‚ 9691.6953125 disk.dm-0.out‚60970.98828125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0703125 disk.sda.in‚ 0.0703125 disk.rbd0.out‚0 disk.sda.out‚ 303.4296875( proc.memory.percent‚17.6177978515625* disk./.usagePercent‚39.460228441052564 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.out‚ 303.4296875 disk.rbd1.in‚506.4375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 9691.6953125 disk.sda2.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2886.5 network.recv‚ 1.255136e+06$ disk./.usageGB‚587.9859199523926" disk.nvme0c0n1.in‚ 9691.6953125% disk.nvme0c0n1.out‚60970.98828125" memory_percent‚88.140869140625 proc.memory.availableMB‚1943 disk.dm-1.out‚ 303.4296875" disk.rbd1.out‚17765.3134765625¼÷딦‚(:‰ ºÄêÒ«Ó­ gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚338.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌY§‚(:Š ºÄêÒ¦Þé® disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 5.508903e+06 disk.rbd0.out‚0 disk.rbd2.out‚29.15966796875) disk./.usagePercent‚39.45998472187997 disk.sda.in‚ 0.0703125 disk.dm-0.in‚9766.25 disk.dm-1.out‚ 305.296875( disk.nvme0c0n1.out‚60981.82080078125 disk.nvme0n1.in‚9766.25 disk.sda.out‚ 305.296875 disk.sda3.out‚ 305.296875 proc.memory.rssMB‚2886.5( proc.memory.percent‚17.6177978515625& disk.nvme0n1.out‚60981.82080078125 disk.sda2.in‚0# disk.dm-0.out‚60981.82080078125 disk.sda1.out‚0$ memory_percent‚88.14091682434082 disk.rbd1.in‚506.4375" disk.rbd1.out‚17765.3134765625$ disk./.usageGB‚587.9822883605957 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 cpu‚25.48295445904299 network.recv‚ 1.260501e+06 disk.nvme0c0n1.in‚9766.25( proc.memory.availableMB‚ 1942.9921875 proc.cpu.threads‚17Ö{dž¨‚(:“ ÉÄêÒž¿™® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«çéuí©‚(:¤ ÉÄêÒ¡²÷® disk.rbd2.out‚29.79248046875$ disk./.usageGB‚587.9779777526855 disk.sda.in‚ 0.0703125 disk.sda.out‚ 307.83984375 disk.sda2.out‚0$ memory_percent‚88.14089298248291( disk.nvme0c0n1.out‚Bq™B60996.98193359375 disk.rbd2.in‚0.015625# disk.dm-0.out‚60996.98193359375 disk.sda3.in‚ 0.0703125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 307.83984375 disk.rbd0.in‚0 network.recv‚ 1.264069e+06& disk.nvme0n1.out‚60996.98193359375" disk.rbd1.out‚17765.4072265625) proc.memory.availableMB‚ 1942.99609375 proc.cpu.threads‚17 disk.sda3.out‚ 307.83984375# disk.nvme0c0n1.in‚ 9767.20703125) disk./.usagePercent‚39.45969543336628 disk.drbd2165.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚2886.5( proc.memory.percent‚17.6177978515625 network.sent‚ 5.513939e+06 disk.dm-1.in‚ 0.0703125 disk.rbd1.in‚506.4375 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 9767.20703125 disk.dm-0.in‚ 9767.20703125 cpu‚25.482601617232856ª. žª‚(:“ ØÄêÒ­ýø« gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“(®!«‚(:– ØÄêÒ·ýì®" disk.rbd1.out‚17765.4658203125$ disk.nvme0n1.out‚61005.248046875 disk.sda.in‚ 0.0703125 disk.sda.out‚ 308.296875 network.sent‚ 5.519016e+06 disk.sda2.in‚0 disk.rbd2.out‚29.98291015625 proc.memory.rssMB‚2886.5" disk.nvme0c0n1.in‚ 9768.0390625& disk.nvme0c0n1.out‚61005.248046875 disk.rbd0.out‚0 disk.drbd2165.out‚0( proc.memory.percent‚17.6177978515625 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.0703125 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 9768.0390625 network.recv‚ 1.267488e+06* disk./.usagePercent‚39.459718218036826$ disk./.usageGB‚587.9783172607422! disk.dm-0.out‚61005.248046875 disk.sda1.in‚0 cpu‚25.482353391905512 disk.dm-1.in‚ 0.0703125 disk.dm-0.in‚ 9768.0390625 disk.drbd2165.in‚0$ memory_percent‚88.14105987548828& proc.memory.availableMB‚ 1942.96875 disk.dm-1.out‚ 308.296875 disk.sda3.out‚ 308.296875 disk.sda2.out‚0?Ó(Ò”¬‚(:‰ çÄêÒ¹¹š¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£P¸.­‚(:£ çÄêÒÆÝ®® disk.dm-1.in‚ 0.0703125) disk./.usagePercent‚39.45973869863956$ disk./.usageGB‚587.9786224365234 disk.sda3.in‚ 0.0703125 disk.sda2.in‚0" disk.rbd1.out‚17765.5439453125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 308.83203125 disk.sda1.in‚0 cpu‚25.48201086314227& disk.nvme0n1.out‚61013.00634765625 disk.sda2.out‚0 disk.dm-0.in‚ 9768.0390625$ memory_percent‚88.13927173614502 proc.memory.rssMB‚2886.5 disk.rbd2.in‚0.015625# disk.dm-0.out‚61013.00634765625 disk.sda1.out‚0 disk.sda3.out‚ 308.83203125 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.sent‚ 5.524623e+06 network.recv‚ 1.270877e+06( disk.nvme0c0n1.out‚61013.00634765625 disk.nvme0n1.in‚ 9768.0390625 disk.drbd2165.in‚0( proc.memory.percent‚17.6177978515625" disk.nvme0c0n1.in‚ 9768.0390625 disk.rbd1.in‚ 506.4921875 disk.rbd2.out‚29.98291015625 disk.dm-1.out‚ 308.83203125 disk.sda.in‚ 0.0703125) proc.memory.availableMB‚ 1943.261718758Ès[2®- _wandbruntime‚4974  _runtime‚4974Žîsɯà % _timestamp‚1.7843247212326634e+09 _runtime‚4974.607390914 _step‚243 ' lr-AdamW/pg1‚0.00026234698630013737 ' lr-AdamW/pg2‚0.00026234698630013737 trainer/global_step‚2999°„¢Î°È _runtime‚4974.607390914 _step‚243 ' lr-AdamW/pg1‚0.00026234698630013737 ' lr-AdamW/pg2‚0.00026234698630013737 trainer/global_step‚2999 % _timestamp‚1.7843247212326634e+09ó ¸Ax2±- _wandbruntime‚4974  _runtime‚4974I˜ï'²éO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚860885562621952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25546413 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863704685700608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7695625 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚860513210888192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843247212338903e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25964272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25546413 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3637890351104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863328880974570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863327510946280 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7695630 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236625615942656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚860885562621952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8520098 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863704685700608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚860885536448512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4974.608649781 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚2999 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3637855814656 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚860885536448512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25546407 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25546407 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2820494641664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232988128219136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25954299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2820494641664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8520347 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232987760128000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820»m†ô'³îO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25546413 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3637890351104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863328880974570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863327510946280 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7695630 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236625615942656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚860885562621952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8520098 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863704685700608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚860885536448512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4974.608649781 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚2999 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3637855814656 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚860885536448512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25546407 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25546407 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2820494641664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232988128219136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25954299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2820494641664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8520347 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232987760128000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚860885562621952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25546413 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863704685700608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7695625 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚860513210888192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843247212338903e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25964272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584ôì*Šµ2´- _wandbruntime‚4975  _runtime‚4975‚i‡ÒµóM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2820074287616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚245 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2821419684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863607686219104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25554619 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16221220 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3639086697472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚860791092666368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236704298703872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25962777 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16220971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25554613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236704701324288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7698168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8523047 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843247226194074e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860791067238400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233065614626816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25554619 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863983611466240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25972764 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25972764 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚4975.994271529 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚861163563352064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚861163563352064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16221220 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3639086697472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚860791092666368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236704298703872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25962777 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16220971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25554613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236704701324288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7698168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8523047 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843247226194074e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860791067238400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233065614626816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25554619 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863983611466240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6õèA³É2·- _wandbruntime‚4976  _runtime‚4976(o¼©¸þ 2loss/FBt1/assignment_loss‚0.022952804610163868 1loss/SRqqt1/detection_loss‚0.7265529258854871 _runtime‚4976.080545258 _step‚246 /loss/FRt2/assignment_loss‚2.141330288161694 /loss/FRt1/detection_loss‚1.6748168624751747 /loss/FRt2/detection_loss‚0.6260243416986391 /loss/FBt2/detection_loss‚0.8097543680508222 $loss/total_loss‚9.84725180641142 epoch‚7 0loss/FBt2/assignment_loss‚0.2021685656322794 0loss/FBt1/detection_loss‚0.46013088956462234 % _timestamp‚1.7843247227061858e+09 /loss/FRt1/assignment_loss‚0.932190699498062 2loss/SRqqt1/assignment_loss‚0.5532447423644721 2loss/SRqqt2/assignment_loss‚0.6658704535926396 0loss/SRqqt2/detection_loss‚1.032214864877363 trainer/global_step‚2999µ×Fø ¹ƒ 2loss/SRqqt1/assignment_loss‚0.5532447423644721 2loss/SRqqt2/assignment_loss‚0.6658704535926396 0loss/SRqqt2/detection_loss‚1.032214864877363 trainer/global_step‚2999 2loss/FBt1/assignment_loss‚0.022952804610163868 1loss/SRqqt1/detection_loss‚0.7265529258854871 _runtime‚4976.080545258 _step‚246 /loss/FRt2/assignment_loss‚2.141330288161694 /loss/FRt1/detection_loss‚1.6748168624751747 /loss/FRt2/detection_loss‚0.6260243416986391 /loss/FBt2/detection_loss‚0.8097543680508222 $loss/total_loss‚9.84725180641142 epoch‚7 0loss/FBt2/assignment_loss‚0.2021685656322794 0loss/FBt1/detection_loss‚0.46013088956462234 % _timestamp‚1.7843247227061858e+09 /loss/FRt1/assignment_loss‚0.932190699498062öÝÝ:’º‚(:‡ öÄêÒÖÜ’­ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚345.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼»‚(:“ öÄêÒ¼Ûå­ disk.sda2.in‚0 disk.dm-0.out‚61020.69921875 disk.drbd2165.in‚0 disk.dm-1.out‚ 309.40625 disk.sda3.out‚ 309.40625! disk.nvme0n1.in‚ 9866.58203125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.0703125 network.sent‚ 5.529111e+06% disk.nvme0c0n1.out‚61020.69921875# disk.nvme0n1.out‚61020.69921875 disk.sda.in‚ 0.0703125* disk./.usagePercent‚39.459740746699836" disk.rbd1.out‚17765.5439453125 disk.rbd2.in‚0.015625 disk.sda.out‚ 309.40625$ memory_percent‚88.14072608947754 proc.memory.rssMB‚2886.5( proc.memory.availableMB‚ 1943.0234375 cpu‚25.481771202345403 disk.rbd1.in‚ 506.4921875 disk.rbd2.out‚ 30.3935546875 disk.sda2.out‚0 network.recv‚ 1.27348e+06$ disk./.usageGB‚587.9786529541016# disk.nvme0c0n1.in‚ 9866.58203125( proc.memory.percent‚17.6177978515625 disk.sda3.in‚ 0.0703125 disk.rbd0.in‚0 disk.dm-0.in‚ 9866.58203125 disk.sda1.out‚0 proc.cpu.threads‚17½Y‚>”¼‚(:‰ …ÅêÒ”ÑÏ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ü–+½‚(:  …ÅêÒ‰ Û­ network.sent‚ 5.591415e+06 disk.sda3.out‚ 309.7734375& disk.nvme0n1.out‚61033.25048828125 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda2.in‚0 disk.dm-0.in‚ 9867.4140625 proc.cpu.threads‚17 disk.sda.out‚ 309.7734375 disk.rbd2.out‚ 30.3935546875 proc.memory.rssMB‚2886.5 network.recv‚ 1.278762e+06 disk.sda3.in‚ 0.0703125" disk.nvme0c0n1.in‚ 9867.4140625 disk.nvme0n1.in‚ 9867.4140625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.0703125 disk.rbd1.in‚ 506.4921875" disk.rbd1.out‚17765.5751953125# disk.dm-0.out‚61033.25048828125 disk.dm-1.out‚ 309.7734375( proc.memory.percent‚17.6177978515625$ memory_percent‚88.14036846160889) proc.memory.availableMB‚ 1943.08203125$ disk./.usageGB‚587.9778366088867 disk.rbd0.in‚0 cpu‚25.481428020500154) disk./.usagePercent‚39.45968596108751( disk.nvme0c0n1.out‚61033.25048828125 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0ÎèRœ¾‚(:‘ ”ÅêÒú¹’­ gpu.0.gpu‚9.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H¶¼Æ¿‚(: ”ÅêÒ¯% disk.nvme0n1.out‚61037.5146484375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚310.3125$ memory_percent‚88.13896179199219 disk.rbd1.in‚ 506.4921875 disk.rbd2.out‚ 30.7021484375 disk.drbd2165.out‚0 disk.dm-1.out‚310.3125* disk./.usagePercent‚39.459656008206004 disk.sda.out‚310.3125 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 9867.4140625" disk.nvme0c0n1.in‚ 9867.4140625( proc.memory.percent‚17.6177978515625 proc.cpu.threads‚17 proc.memory.rssMB‚2886.5 cpu‚25.48126740700897 network.sent‚ 5.596987e+06% proc.memory.availableMB‚ 1943.3125 disk.nvme0n1.in‚ 9867.4140625 disk.rbd2.in‚0.015625$ disk./.usageGB‚587.9773902893066 disk.sda.in‚ 0.0703125" disk.rbd1.out‚17765.7998046875 disk.rbd0.in‚0" disk.dm-0.out‚61037.5146484375 disk.sda1.in‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125' disk.nvme0c0n1.out‚61037.5146484375 network.recv‚ 1.282034e+06×zÇ”À‚(:‰ £ÅêÒÁ­ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.217' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓQ½€,Á‚(:¡ £ÅêÒÁžò¯$ memory_percent‚88.13900947570801 disk.drbd2165.in‚0& disk.nvme0n1.out‚61054.34814453125 disk.sda.in‚ 0.0703125 disk.sda1.in‚0( proc.memory.availableMB‚ 1943.3046875 proc.memory.rssMB‚2886.5 proc.cpu.threads‚17 disk.sda3.in‚ 0.0703125 disk.sda3.out‚ 310.5859375( disk.nvme0c0n1.out‚61054.34814453125 disk.sda.out‚ 310.5859375 disk.rbd0.out‚0 network.sent‚ 5.601555e+06# disk.nvme0c0n1.in‚ 9868.37109375 disk.sda2.in‚0 disk.dm-0.in‚ 9868.37109375) disk./.usagePercent‚39.45992968026011! disk.nvme0n1.in‚ 9868.37109375 disk.sda2.out‚0# disk.dm-0.out‚61054.34814453125 cpu‚25.48092132603538 network.recv‚ 1.285407e+06 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 310.5859375" disk.rbd1.out‚17765.7998046875 disk.sda1.out‚0( proc.memory.percent‚17.6177978515625$ disk./.usageGB‚587.9814682006836 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 30.7021484375¹‚µ©’‚(:‡ ²ÅêÒæ š® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ô0$2Â(:§ ²ÅêÒ½“†¯ disk.rbd0.in‚0 disk.rbd2.out‚30.82080078125( proc.memory.availableMB‚ 1943.3046875 disk.sda3.out‚ 313.64453125 disk.dm-0.in‚ 9984.59765625 disk.sda1.in‚0( proc.memory.percent‚17.6177978515625 proc.cpu.threads‚17 network.sent‚ 5.606877e+06$ memory_percent‚88.13900947570801 proc.memory.rssMB‚2886.5 disk.sda3.in‚ 0.0703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 9984.59765625# disk.dm-0.out‚61074.34814453125 disk.sda1.out‚0 network.recv‚ 1.288946e+06 disk.sda.out‚ 313.64453125 disk.sda2.out‚0 disk.rbd0.out‚0 cpu‚25.480641107034277 disk.dm-1.out‚ 313.64453125# disk.nvme0c0n1.in‚ 9984.59765625( disk.nvme0c0n1.out‚61074.34814453125 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875* disk./.usagePercent‚39.460107605496404& disk.nvme0n1.out‚61074.34814453125 disk.sda.in‚ 0.0703125" disk.rbd1.out‚17765.8623046875 disk.rbd2.in‚0.015625$ disk./.usageGB‚587.9841194152832 disk.dm-1.in‚ 0.0703125̯°Ž2Ä- _wandbruntime‚5054  _runtime‚5054ʉ³ÉÅà ' lr-AdamW/pg2‚0.00026189858624920987 trainer/global_step‚3049 % _timestamp‚1.7843248013640757e+09 _runtime‚5054.738742432 _step‚247 ' lr-AdamW/pg1‚0.00026189858624920987[ºØÎÆÈ _step‚247 ' lr-AdamW/pg1‚0.00026189858624920987 ' lr-AdamW/pg2‚0.00026189858624920987 trainer/global_step‚3049 % _timestamp‚1.7843248013640757e+09 _runtime‚5054.738742432÷¸Ø)Ä2Ç- _wandbruntime‚5054  _runtime‚5054*Ò—Hï'ÈéO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26390024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚874785599127552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25956713 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866810572288 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8657514 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236862448603136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚877269206996478 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2860562599164 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚874785572954112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7822780 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3697380349952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16480289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚877267836968188 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚874785599127552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2861907199486 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52346737 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚877651038136832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865465182720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25956707 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7822775 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874407274369024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26380051 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865465182720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240559460861952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3697414886400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874407299796992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26390024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5054.740009223 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚874785572954112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866810572288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16480547 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236862080512000 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175ÏkÉ_uÉîO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336758 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_byt¼2µi$es.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚877651038136832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865465182720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25956707 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7822775 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874407274369024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26380051 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865465182720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240559460861952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3697414886400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874407299796992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26390024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5054.740009223 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚874785572954112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866810572288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16480547 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236862080512000 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26390024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚874785599127552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25956713 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866810572288 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8657514 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236862448603136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚877269206996478 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2860562599164 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚874785572954112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7822780 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3697380349952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16480289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚877267836968188 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚874785599127552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2861907199486 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52346737 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚5056.135464457 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7825323 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26398516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25964919 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8660175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚877931335472640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875063573684224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866390218240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚875063599857664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26388529 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25964919 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚877929963902464 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16485493 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0^ŸM%þ&ÎøM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26388529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240638533021696 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚877546642212610 % _timestamp‚1.7843248027605264e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236939566919680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚5056.135464457 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7825323 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26398516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25964919 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8660175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚877931335472640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875063573684224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866390218240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚875063599857664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26388529 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25964919 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚877929963902464 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16485493 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875063599857664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867735614976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8660425 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736ù¢A2Ï- _wandbruntime‚5056  _runtime‚5056¬#òÐþ .loss/FRt2/assignment_loss‚2.05945196841461 2loss/SRqqt2/assignment_loss‚0.5766766263921341 &loss/total_loss‚10.179502653599016 epoch‚7 _runtime‚5056.217628915 _step‚250 1loss/FBt1/assignment_loss‚0.07449458388571242 0loss/FBt2/assignment_loss‚0.3238727301559915 trainer/global_step‚3049 /loss/FRt1/detection_loss‚1.5601254874363888 1loss/SRqqt1/detection_loss‚0.6188338435916092 /loss/FBt2/detection_loss‚1.5414328094394434 0loss/FRt1/assignment_loss‚0.8076318168362906 2loss/SRqqt1/assignment_loss‚0.5724997314427817 /loss/FRt2/detection_loss‚0.5975086361782325 1loss/SRqqt2/detection_loss‚1.0799403457639876 0loss/FBt1/detection_loss‚0.36703407406183536 # _timestamp‚1.78432480284303e+09H4õ у # _timestamp‚1.78432480284303e+09 .loss/FRt2/assignment_loss‚2.05945196841461 2loss/SRqqt2/assignment_loss‚0.5766766263921341 &loss/total_loss‚10.179502653599016 epoch‚7 _runtime‚5056.217628915 _step‚250 1loss/FBt1/assignment_loss‚0.07449458388571242 0loss/FBt2/assignment_loss‚0.3238727301559915 trainer/global_step‚3049 /loss/FRt1/detection_loss‚1.5601254874363888 1loss/SRqqt1/detection_loss‚0.6188338435916092 /loss/FBt2/detection_loss‚1.5414328094394434 0loss/FRt1/assignment_loss‚0.8076318168362906 2loss/SRqqt1/assignment_loss‚0.5724997314427817 /loss/FRt2/detection_loss‚0.5975086361782325 1loss/SRqqt2/detection_loss‚1.0799403457639876 0loss/FBt1/detection_loss‚0.36703407406183536ú±§R·Ò‚(:‰ ÐÅêÒ†Ëä« gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.92ítp›~76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V#7Z Ó‚(:• ÐÅêÒ®ý¡®) disk./.usagePercent‚39.46011886982791 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 506.4921875% proc.memory.availableMB‚ 1943.0625 proc.cpu.threads‚17 disk.sda2.out‚0 network.sent‚ 5.664296e+06 disk.sda1.in‚0 disk.sda.out‚ 319.78515625 disk.rbd0.out‚0$ memory_percent‚88.14048767089844 disk.dm-1.out‚ 319.78515625 disk.sda3.in‚ 0.0703125 disk.sda3.out‚ 319.78515625 disk.dm-1.in‚ 0.0703125 cpu‚25.480125448788726$ disk./.usageGB‚587.9842872619629 disk.dm-0.in‚ 9986.38671875$ disk.nvme0c0n1.out‚ 61121.9609375" disk.nvme0n1.out‚ 61121.9609375" disk.rbd1.out‚17765.9248046875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 network.recv‚ 1.297125e+06 disk.dm-0.out‚ 61121.9609375 disk.drbd2165.out‚0( proc.memory.percent‚17.6177978515625# disk.nvme0c0n1.in‚ 9986.38671875! disk.nvme0n1.in‚ 9986.38671875 disk.sda.in‚ 0.0703125 disk.sda2.in‚0 disk.rbd2.out‚ 31.611328125 proc.memory.rssMB‚2886.5s¶³’”Ô‚(:‰ ßÅêÒ––ÿ« gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ã½3Õ‚(:¨ ßÅêÒÅîÕ® network.recv‚ 1.302368e+06 disk.sda3.in‚ 0.0703125 disk.rbd1.in‚ 506.4921875" disk.dm-0.out‚61133.4033203125 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 321.86328125 cpu‚25.479788204276943$ disk./.usageGB‚587.9811859130859' disk.nvme0c0n1.out‚61133.4033203125% disk.nvme0n1.out‚61133.4033203125 disk.dm-1.in‚ 0.0703125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2886.50390625) proc.memory.percent‚17.61782169342041! disk.nvme0n1.in‚ 9986.38671875 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 31.611328125 disk.sda3.out‚ 321.86328125 disk.sda.out‚ 321.86328125" disk.rbd1.out‚17765.9248046875) proc.memory.availableMB‚ 1942.61328125 proc.cpu.threads‚17) disk./.usagePercent‚39.45991073570257# disk.nvme0c0n1.in‚ 9986.38671875 disk.sda.in‚ 0.0703125 disk.rbd0.in‚0 disk.sda1.out‚0 network.sent‚ 5.71722e+06 disk.dm-0.in‚ 9986.38671875# memory_percent‚88.1432294845581Y'?”Ö‚(:‰ îÅêÒ»Ëï« gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚327.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.4694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 a«`'ׂ(:œ îÅêÒ廬® disk.drbd2165.in‚0 disk.dm-1.out‚ 322.80078125 disk.sda.in‚ 0.0703125 disk.rbd0.out‚0 cpu‚25.479606706010877 disk.rbd0.in‚0 disk.rbd2.in‚0.015625! disk.nvme0c0n1.in‚ 10140.40625 network.sent‚ 5.722379e+06$ disk./.usageGB‚587.9807624816895 disk.dm-0.out‚61140.62890625 disk.sda2.out‚0) disk./.usagePercent‚39.45988231886627 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125 disk.sda3.out‚ 322.80078125 disk.nvme0n1.in‚ 10140.40625 proc.cpu.threads‚17 network.recv‚ 1.305943e+06 disk.rbd2.out‚31.65478515625% disk.nvme0c0n1.out‚61140.62890625" disk.rbd1.out‚17765.9873046875# memory_percent‚88.1429672241211# proc.memory.rssMB‚ 2886.50390625 disk.dm-0.in‚ 10140.40625 disk.sda2.in‚0& proc.memory.availableMB‚ 1942.65625) proc.memory.percent‚17.61782169342041 disk.sda1.in‚0# disk.nvme0n1.out‚61140.62890625 disk.sda.out‚ 322.80078125 disk.rbd1.in‚ 506.4921875 ¡}žØ‚(:“ ýÅêÒô¾À¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.287' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.05739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00“Ñ7Ù‚(:¬ ýÅêÒç±Æ®) disk./.usagePercent‚39.45992404809435 disk.rbd1.in‚ 506.4921875" disk.rbd1.out‚17765.9873046875 disk.rbd0.in‚0 disk.dm-0.in‚10141.36328125 disk.sda1.in‚0" disk.nvme0n1.in‚10141.36328125 disk.sda3.out‚ 323.2421875& disk.nvme0n1.out‚61151.16845703125) proc.memory.percent‚17.61782169342041$ disk./.usageGB‚587.9813842773438 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 323.2421875 disk.sda3.in‚ 0.0703125( disk.nvme0c0n1.out‚61151.16845703125" memory_percent‚88.142991065979 network.sent‚ 5.727608e+06 disk.sda.out‚ 323.2421875 disk.rbd2.out‚31.65478515625 disk.sda2.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚10141.36328125 cpu‚25.47932364326623 disk.sda.in‚ 0.0703125 disk.rbd0.out‚0# disk.dm-0.out‚61151.16845703125 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1942.65234375# proc.memory.rssMB‚ 2886.50390625 disk.rbd2.in‚0.015625 network.recv‚ 1.309471e+06 disk.sda2.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17Kf›ÉžÚ‚(:“ ŒÆêÒ²¨¡¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èë5Û‚(:ª ŒÆêÒ…Íú­ disk.drbd2165.out‚0) proc.memory.availableMB‚ 1942.66015625 disk.sda.in‚ 0.0703125 disk.dm-1.in‚ 0.0703125& disk.nvme0n1.out‚61155.26806640625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2886.50390625$ disk.nvme0c0n1.in‚10141.36328125( disk.nvme0c0n1.out‚61155.26806640625" disk.nvme0n1.in‚10141.36328125 disk.rbd0.out‚0 disk.dm-1.out‚ 324.109375) disk./.usagePercent‚39.45972538624778 disk.sda3.in‚ 0.0703125 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚88.14294338226318 network.recv‚ 1.313067e+06 disk.sda3.out‚ 324.109375 disk.sda.out‚ 324.109375 disk.sda2.in‚0" disk.rbd1.out‚17766.0498046875) proc.memory.percent‚17.61782169342041 network.sent‚ 5.732221e+06 disk.rbd2.in‚0.015625 disk.sda1.out‚0 cpu‚25.47914745920961 proc.cpu.threads‚17$ disk./.usageGB‚587.9784240722656 disk.rbd2.out‚ 31.7177734375 disk.dm-0.in‚10141.36328125# disk.dm-0.out‚61155.26806640625²e2Ü- _wandbruntime‚5134  _runtime‚5134æï“ÆÝÀ trainer/global_step‚3099 % _timestamp‚1.7843248812145615e+09 _runtime‚ 5134.58913128 _step‚251 & lr-AdamW/pg1‚0.0002614421663281631 & lr-AdamW/pg2‚0.0002614421663281631¹÷a3ËÞÅ & lr-AdamW/pg1‚0.0002614421663281631 & lr-AdamW/pg2‚0.0002614421663281631 trainer/global_step‚3099 % _timestamp‚1.7843248812145615e+09 _runtime‚ 5134.58913128 _step‚251ûlÅüK2ß- _wandbruntime‚5134  _runtime‚5134KšUî'àèO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚252 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26805803 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8794716 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888685635633152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26815776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2911781113344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891598762136064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888685609459712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26367013 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26367013 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244493731689472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16744646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚891598762136064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2911781113344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182789 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240736768987136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891597390573056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244493331159040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240736400896000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚888301363277824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26815776 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26805803 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888685635633152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888685609459712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8794470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26367007 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _runtime‚5134.590455049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2913126502912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚888301388705792 $ _timestamp‚1.784324881215713e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚891597390573056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7949930 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3756930263040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7949925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3756962702336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚891209533018386 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚891208162990096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2906799712272zÇvçó'áíO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2913126502912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16744395 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2911781113344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182789 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240736768987136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891597390573056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244493331159040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240736400896000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚888301363277824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26815776 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26805803 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888685635633152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888685609459712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8794470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26367007 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _runtime‚5134.590455049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2913126502912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚888301388705792 $ _timestamp‚1.784324881215713e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚891597390573056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7949930 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3756930263040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7949925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3756962702336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚891209533018386 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚891208162990096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2906799712272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚252 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26805803 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8794716 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888685635633152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26815776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2911781113344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891598762136064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888685609459712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26367013 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26367013 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244493731689472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16744646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚891598762136064 B¯[>²ãóM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888963636363264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3758152904704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8797226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891877687908864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3758120472576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891488338262920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26375219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2909067778952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26375213 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888963610189824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16749694 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240813887303680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26375219 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26814281 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚888579270483968ß !µþ&äøM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16749694 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240813887303680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26375219 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26814281 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚888579270483968 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888963636363264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3758152904704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8797226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891877687908864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3758120472576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891488338262920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26375219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2909067778952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26375213 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888963610189824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736ýs…«y2å- _wandbruntime‚5136  _runtime‚5136º*Þæÿ /loss/FBt1/detection_loss‚0.8180834190025925 %loss/total_loss‚9.639375858311425 .loss/FRt2/assignment_loss‚1.92125810544984 /loss/FRt2/detection_loss‚0.6459619551317747 trainer/global_step‚3099 2loss/SRqqt1/assignment_loss‚0.5882070266780227 2loss/SRqqt2/assignment_loss‚0.6957131212522986 1loss/FBt2/assignment_loss‚0.09880042808025323 1loss/SRqqt1/detection_loss‚0.6523813967356904 % _timestamp‚1.7843248826874664e+09 _step‚254 0loss/FRt1/assignment_loss‚0.9345009410466264 /loss/FRt1/detection_loss‚1.5964341786935607 1loss/SRqqt2/detection_loss‚1.0038969026865892 /loss/FBt2/detection_loss‚0.6471534653594999 epoch‚7 _runtime‚5136.061812546 1loss/FBt1/assignment_loss‚0.03698491819467597ù>¶è ç„ 1loss/SRqqt2/detection_loss‚1.0038969026865892 /loss/FBt2/detection_loss‚0.6471534653594999 epoch‚7 _runtime‚5136.061812546 1loss/FBt1/assignment_loss‚0.03698491819467597 /loss/FBt1/detection_loss‚0.8180834190025925 %loss/total_loss‚9.639375858311425 .loss/FRt2/assignment_loss‚1.92125810544984 /loss/FRt2/detection_loss‚0.6459619551317747 trainer/global_step‚3099 2loss/SRqqt1/assignment_loss‚0.5882070266780227 2loss/SRqqt2/assignment_loss‚0.6957131212522986 1loss/FBt2/assignment_loss‚0.09880042808025323 1loss/SRqqt1/detection_loss‚0.6523813967356904 % _timestamp‚1.7843248826874664e+09 _step‚254 0loss/FRt1/assignment_loss‚0.9345009410466264 /loss/FRt1/detection_loss‚1.5964341786935607þú£âŸè‚(:” ›ÆêÒâÈ¢¯ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.616' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.923199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×àg*é‚(:Ÿ ›ÆêÒÕÕ˜° cpu‚25.47879821126351 disk.dm-1.in‚ 0.0703125 proc.cpu.threads‚17 disk.rbd0.out‚0* disk./.usagePercent‚39.459791692199154& disk.nvme0n1.out‚61166.35888671875" disk.rbd1.out‚17766.0498046875# disk.dm-0.out‚61166.35888671875 disk.sda.out‚324.375 network.sent‚ 5.737977e+06$ disk./.usageGB‚587.9794120788574 disk.dm-0.in‚ 10142.1953125 disk.sda3.in‚ 0.0703125! disk.nvme0n1.in‚ 10142.1953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 network.recv‚ 1.316872e+06 disk.rbd2.in‚0.015625 disk.sda1.out‚0$ memory_percent‚88.14284801483154) proc.memory.percent‚17.61782169342041 disk.rbd2.out‚ 31.7177734375 disk.dm-1.out‚324.375 disk.sda.in‚ 0.0703125) proc.memory.availableMB‚ 1942.67578125# proc.memory.rssMB‚ 2886.50390625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚324.375# disk.nvme0c0n1.in‚ 10142.1953125( disk.nvme0c0n1.out‚61166.35888671875£€Ã”ê‚(:‰ ªÆêÒª›í« gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.274' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.8548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õûZC9ë‚(:® ªÆêÒ¬ƒ»®" disk.rbd1.out‚17766.1123046875 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0703125# proc.memory.rssMB‚ 2886.50390625 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚61174.07666015625 disk.sda.in‚ 0.0703125 disk.dm-0.in‚ 10225.4296875 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1942.67578125) proc.memory.percent‚17.61782169342041 network.sent‚ 5.800493e+06 disk.sda.out‚ 324.67578125 disk.rbd2.out‚ 31.7802734375 network.recv‚ 1.322251e+06 disk.dm-1.out‚ 324.67578125( disk.nvme0c0n1.out‚61174.07666015625 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda3.out‚ 324.67578125# disk.nvme0c0n1.in‚ 10225.4296875 proc.cpu.threads‚17) disk./.usagePercent‚39.45980065246285! disk.nvme0n1.in‚ 10225.4296875 disk.sda2.in‚0# disk.dm-0.out‚61174.07666015625 disk.dm-1.in‚ 0.0703125$ memory_percent‚88.14284801483154 cpu‚25.478636113569628$ disk./.usageGB‚587.9795455932617 disk.sda2.out‚0GGɹŸì‚(:” ¹ÆêÒˆù±¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 -Ž[1í‚(:¦ ¹ÆêÒÍå”® disk.rbd2.out‚ 33.7646484375# disk.dm-0.out‚61180.60107421875 disk.sda1.out‚0 disk.sda.in‚ 0.0703125 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0703125 disk.drbd2165.out‚0$ disk./.usageGB‚587.9798126220703 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚61180.60107421875 network.sent‚ 5.806081e+06 network.recv‚ 1.326004e+06 disk.dm-1.out‚ 325.15625 disk.sda2.in‚0" disk.rbd1.out‚17766.1123046875 cpu‚25.478220007259242 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0 disk.dm-0.in‚ 10225.4296875 disk.sda3.out‚ 325.15625$ memory_percent‚88.14270496368408 proc.cpu.threads‚17& disk.nvme0n1.out‚61180.60107421875 disk.sda.out‚ 325.15625) proc.memory.availableMB‚ 1942.69921875# proc.memory.rssMB‚ 2886.50390625 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.0703125! disk.nvme0n1.in‚ 10225.4296875) proc.memory.percent‚17.61782169342041* disk./.usagePercent‚39.459818572990244 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 10225.4296875çÓê’î‚(:‡ ÈÆêÒ…ꮬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚330.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O,88ï‚(:­ ÈÆêÒò†²®" disk.nvme0n1.in‚10226.26171875 disk.rbd2.in‚0.015625 disk.dm-0.in‚10226.26171875 disk.dm-1.in‚ 0.0703125 disk.sda3.out‚ 327.7421875 cpu‚25.47804068388728 disk.sda.in‚ 0.0703125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2886.50390625& disk.nvme0n1.out‚61205.71337890625 disk.sda.out‚ 327.7421875 disk.rbd0.out‚0$ memory_percent‚88.14272880554199( proc.memory.availableMB‚ 1942.6953125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.0703125" disk.rbd1.out‚17766.1748046875 network.sent‚ 5.810854e+06$ disk.nvme0c0n1.in‚10226.26171875 disk.drbd2165.out‚0 disk.rbd2.out‚33.88330078125 disk.dm-1.out‚ 327.7421875 disk.sda1.out‚0 disk.rbd1.in‚ 506.4921875$ disk./.usageGB‚587.9891510009766 disk.sda2.in‚0# disk.dm-0.out‚61205.71337890625 disk.sda1.in‚0) proc.memory.percent‚17.61782169342041) disk./.usagePercent‚39.46044527943406( disk.nvme0c0n1.out‚61205.71337890625 network.recv‚ 1.329587e+06=›g§Ÿð‚(:” ×ÆêÒ£úç¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MåñÖ-ñ‚(:¢ ×ÆêÒóȼ° disk.sda.in‚ 0.0703125) proc.memory.percent‚17.61782169342041$ disk./.usageGB‚587.9904708862305 disk.dm-1.out‚ 329.328125 disk.rbd1.in‚ 506.4921875 disk.dm-0.in‚10226.26171875 disk.sda3.in‚ 0.0703125" disk.rbd1.out‚17766.1748046875 network.recv‚ 1.333245e+06) disk./.usagePercent‚39.46053385804091 disk.rbd0.out‚0 disk.dm-0.out‚61225.70703125 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.out‚61225.70703125 disk.sda.out‚ 329.328125 disk.sda2.in‚0 disk.rbd2.in‚0.015625) proc.memory.availableMB‚ 1943.09765625" disk.nvme0n1.in‚10226.26171875 disk.rbd2.out‚33.88330078125# proc.memory.rssMB‚ 2886.50390625 disk.dm-1.in‚ 0.0703125 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚88.14027309417725 cpu‚25.477748010981163 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda3.out‚ 329.328125$ disk.nvme0c0n1.in‚10226.26171875% disk.nvme0c0n1.out‚61225.70703125 disk.rbd0.in‚0 network.sent‚ 5.81596e+06ƒÌRM2ò- _wandbruntime‚5213  _runtime‚5213¥M¨Éóà % _timestamp‚1.7843249598273141e+09 _runtime‚5213.201896248 _step‚255 ' lr-AdamW/pg1‚0.00026097775585199587 ' lr-AdamW/pg2‚0.00026097775585199587 trainer/global_step‚3149ĵêÎôÈ _step‚255 ' lr-AdamW/pg1‚0.00026097775585199587 ' lr-AdamW/pg2‚0.00026097775585199587 trainer/global_step‚3149 % _timestamp‚1.7843249598273141e+09 _runtime‚5213.201896248ÿx 8W2õ- _wandbruntime‚5213  _runtime‚5213˜ÒÀ·÷öéO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚902195477614592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902585672138752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26777313 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚905545114572288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902585645965312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244611089371136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚902195452186624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248427807391744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244610721280000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3816718020608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248427406861312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2953036825380 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8077080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843249598284056e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2959442433536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27241528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17008533 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8077075 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚905149859040294 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17008776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902585672138752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚905543743009280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902585645965312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26777313 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26777307 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2954381425702 _runtime‚5213.203125689 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚905543743009280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8931458 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2958097043968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peoíÕžø ak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚905545114572288 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚256 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248427406861312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2953036825380 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8077080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843249598284056e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2959442433536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27241528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17008533 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8077075 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚905149859040294 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17008776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902585672138752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚905543743009280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902585645965312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26777313 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26777307 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2954381425702 _runtime‚5213.203125689 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚905543743009280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8931458 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2958097043968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚905545114572288 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚256 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27250020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2955304892060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902863646695424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17013816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚905824040345088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8079618 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2953960291626 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244688207687680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚905427294256426 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2960367476224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚902473359392768 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3149 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26785513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2959022079488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚905824040345088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26785519 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248506487917568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035539 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025546 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17014052 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27250020 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3817879706624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434SJóÆ&úøM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3817912138752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚902473333964800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚905428664284828 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54025546 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27240033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚5214.599796919 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2959022079488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248506087394304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8934429 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27250020 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3817879706624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8934198 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27250020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2955304892060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902863646695424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17013816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚905824040345088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8079618 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2953960291626 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244688207687680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚905427294256426 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2960367476224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚902473359392768 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3149 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26785513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2959022079488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚905824040345088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26785519 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248506487917568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035539 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025546 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17014052 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 Bÿ‚(:³ æÆêÒÕ™¯ disk.rbd2.in‚0.015625 disk.dm-0.in‚10289.89453125 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2764.8671875& disk.nvme0n1.out‚61240.28564453125 disk.sda.in‚ 0.0703125 disk.rbd1.in‚ 506.4921875 proc.cpu.threads‚17 network.sent‚ 5.821121e+06 disk.rbd0.in‚0# disk.dm-0.out‚61240.28564453125 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda2.in‚0 network.recv‚ 1.336595e+06$ disk./.usageGB‚587.9918937683105 disk.dm-1.out‚ 330.94140625 disk.sda3.in‚ 0.0703125$ disk.nvme0c0n1.in‚10289.89453125 cpu‚25.477824377345097 disk.rbd0.out‚0( disk.nvme0c0n1.out‚61240.28564453125" disk.nvme0n1.in‚10289.89453125 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 330.94140625" disk.rbd1.out‚17766.2373046875* disk./.usagePercent‚39.460629348851185 disk.rbd2.out‚34.01318359375* proc.memory.percent‚16.875410079956055 disk.sda.out‚ 330.94140625$ memory_percent‚87.66453266143799) proc.memory.availableMB‚ 2021.04296875§9(ᔀ‚(:‰ õÆêÒÙ‘²­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ad:‚(:¯ õÆêÒ“཮ disk.dm-1.out‚ 333.64453125( disk.nvme0c0n1.out‚61259.18115234375 disk.sda.out‚ 333.64453125 disk.rbd0.out‚0 disk.dm-0.in‚ 10290.7265625) proc.memory.availableMB‚ 2021.24609375 network.recv‚ 1.34187e+06! disk.nvme0n1.in‚ 10290.7265625 disk.rbd1.in‚ 506.4921875 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2764.8671875# disk.dm-0.out‚61259.18115234375 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0703125$ memory_percent‚87.66329288482666$ disk./.usageGB‚587.9888458251953 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 333.64453125" disk.rbd1.out‚17766.2373046875 network.sent‚ 5.884138e+06* disk./.usagePercent‚39.460424798831326 disk.sda2.in‚0 proc.cpu.threads‚17 cpu‚25.478370247016667 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 10290.7265625& disk.nvme0n1.out‚61259.18115234375 disk.sda.in‚ 0.0703125* proc.memory.percent‚16.875410079956055 disk.rbd2.out‚34.01318359375 disk.sda3.in‚ 0.0703125Ìqpž‚‚(:“ „ÇêÒÝ»º¬ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.737' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.34740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š­óP9ƒ‚(:® „ÇêÒ”í¬¯ disk.sda3.out‚ 336.53515625* proc.memory.percent‚16.875410079956055 disk.sda2.out‚0 cpu‚25.479228943546516 proc.cpu.threads‚17 disk.sda.out‚ 336.53515625 disk.sda3.in‚ 0.0703125( disk.nvme0c0n1.out‚61267.35693359375& disk.nvme0n1.out‚61267.35693359375 disk.rbd2.in‚0.015625" proc.memory.rssMB‚ 2764.8671875! disk.nvme0n1.in‚ 10290.7265625 disk.sda.in‚ 0.0703125 disk.rbd0.in‚0$ memory_percent‚87.66329288482666) proc.memory.availableMB‚ 2021.24609375 network.recv‚ 1.345051e+06* disk./.usagePercent‚39.459809868734084 disk.sda2.in‚0" disk.rbd1.out‚17766.2998046875 disk.dm-0.in‚ 10290.7265625# disk.dm-0.out‚61267.35693359375 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0703125# disk.nvme0c0n1.in‚ 10290.7265625 disk.rbd0.out‚0 network.sent‚ 5.889259e+06$ disk./.usageGB‚587.9796829223633 disk.rbd2.out‚ 34.037109375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 506.4921875 disk.sda1.in‚0 disk.dm-1.out‚ 336.53515625ÖÝ;´„‚(:’ “ÇêÒ¾“¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.23' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.24600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÅÑ3…‚(:¨ “ÇêÒê¬Ð® disk.rbd0.in‚0( proc.memory.availableMB‚ 2021.2421875" disk.nvme0n1.in‚10291.68359375 disk.sda2.in‚0" disk.rbd1.out‚17766.2998046875 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2764.8671875 cpu‚25.479861950139902 disk.rbd2.out‚ 34.037109375 disk.sda1.out‚0 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 338.54296875$ disk.nvme0n1.out‚61293.330078125 disk.dm-0.in‚10291.68359375! disk.dm-0.out‚61293.330078125 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.in‚ 0.0703125 disk.rbd2.in‚0.015625 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.sda3.out‚ 338.54296875 disk.sda.out‚ 338.54296875 disk.sda1.in‚0$ memory_percent‚87.66331672668457* proc.memory.percent‚16.875410079956055$ disk./.usageGB‚587.9883651733398$ disk.nvme0c0n1.in‚10291.68359375 disk.rbd0.out‚0 network.sent‚ 5.894324e+06 network.recv‚ 1.34875e+06) disk./.usagePercent‚39.46039254188201 disk.sda3.in‚ 0.0703125& disk.nvme0c0n1.out‚61293.330078125vA”†‚(:‰ ¢ÇêÒâ‡Ò¬ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚323.953' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.7906 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¬ª;‡‚(:° ¢ÇêÒ©˜ì® disk.sda1.out‚0" disk.nvme0n1.in‚10350.85546875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2764.8671875& disk.nvme0n1.out‚61311.94091796875 disk.rbd2.in‚0.015625 network.sent‚ 5.899441e+06 network.recv‚ 1.35191e+06$ disk.nvme0c0n1.in‚10350.85546875 disk.rbd0.out‚0 disk.rbd2.out‚34.10009765625 cpu‚25.48057642652808 disk.dm-0.in‚10350.85546875 disk.drbd2165.out‚0 disk.dm-1.out‚ 341.61328125 disk.sda3.in‚ 0.0703125 disk.sda1.in‚0 disk.rbd1.in‚ 506.4921875" disk.rbd1.out‚17766.3623046875$ memory_percent‚87.66491413116455) disk./.usagePercent‚39.46100004776076 disk.dm-1.in‚ 0.0703125 disk.sda3.out‚ 341.61328125 disk.sda2.out‚0$ disk./.usageGB‚587.9974174499512 disk.sda.out‚ 341.61328125) proc.memory.availableMB‚ 2020.98046875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚61311.94091796875 disk.sda.in‚ 0.0703125 disk.sda2.in‚0* proc.memory.percent‚16.875410079956055 proc.cpu.threads‚17# disk.dm-0.out‚61311.94091796875ROõ@2ˆ- _wandbruntime‚5292  _runtime‚52927ᆚljÁ trainer/global_step‚3199 % _timestamp‚1.7843250392413285e+09 _runtime‚5292.615931769 _step‚259 & lr-AdamW/pg1‚0.0002605053846489252 & lr-AdamW/pg2‚0.0002605053846489252±QŠÌŠÆ & lr-AdamW/pg1‚0.0002605053846489252 & lr-AdamW/pg2‚0.0002605053846489252 trainer/global_step‚3199 % _timestamp‚1.7843250392413285e+09 _runtime‚5292.615931769 _step‚259ƒ&8l 2‹- _wandbruntime‚5292  _runtime‚5292.7s'ï'ŒéO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8204225 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843250392424467e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27657307 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252361750362112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27187607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27667280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚916089541095424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916485708644352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚916485682470912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚919490095445504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916485682470912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17272870 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚916089566523392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004412974592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27667280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919090185062202 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3000618538810 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚919490095445504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8204230 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248485041664000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919088815033912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248485409755136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚919491467008512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27187613 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27187607 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5292.617255528 _step‚260 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004412974592 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352½×¦ô'îO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004412974592 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8204225 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843250392424467e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27657307 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252361750362112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27187607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27667280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚916089541095424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916485708644352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚916485682470912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚919490095445504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916485682470912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17272870 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚916089566523392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004412974592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27667280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919090185062202 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3000618538810 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚919490095445504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8204230 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248485041664000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919088815033912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248485409755136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚919491467008512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27187613 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27187607 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5292.617255528 _step‚260„Z™‡¸2Ž- _wandbruntime‚5294  _runtime‚5294™%B¾ òM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000197404734 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919367620278334 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚5294.015273004 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚919368990306736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248562896162816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚916763683201024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27195819 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9071396 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17278164 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27675772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27665785 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784325040640279e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚916367422873600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252440835801088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006683406848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27665785 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9071631 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8206768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006683406848 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3005338010112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27195813 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877907206144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27195813 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877939638272 trainer/global_step‚3199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚916763709374464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚919770392781312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987êwÊý&÷M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8206773 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248562528071680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784325040640279e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚916367422873600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252440835801088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006683406848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27665785 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9071631 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8206768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006683406848 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3005338010112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27195813 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877907206144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27195813 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877939638272 trainer/global_step‚3199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚916763709374464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚919770392781312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000197404734 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919367620278334 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚5294.015273004 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚919368990306736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248562896162816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚916763683201024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27195819 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9071396 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17278164 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27675772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27665785 P&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994839072227478 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9228695631027222 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7657595872879028 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14020821452140808 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚7 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7828420996665955 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3199 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02775241993367672 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933775663375854 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48920994997024536 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14491117000579834 +jet/accuracy_3_of_4‚0.08203098177909851 +particle/specificity‚0.9052958488464355 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5777212977409363 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5794214606285095 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5867123603820801 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941057562828064 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19764435291290283 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_1_of_5‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18072277307510376 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2516794800758362 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5383405685424805 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23400159180164337 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.235098198056221 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959599375724792 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953079223632812 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16502726078033447 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02275569923222065 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1804264783859253 *jet/accuracy_1_of_2‚0.7212826609611511 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48920994997024536 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03192296624183655 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20529361069202423 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950741529464722 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7196440100669861 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7001133561134338 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6891058683395386 particle/accuracy_4_of_5‚1 % _timestamp‚1.7843250681092362e+09 )loss/val_total_loss‚9.356665875195947 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16502726078033447 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17767715454101562 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.643361508846283 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19927656650543213 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6805555820465088 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7validation_average_jet_accuracy‚0.30567559599876404 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5305968523025513 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.9477324485778809 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7865269184112549 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9938897490501404 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10120194405317307 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_5_of_5‚0 _runtime‚5321.484594143 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953079223632812 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14273110032081604 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49780377745628357 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6341850161552429 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4957408308982849 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5871070623397827 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14099855720996857 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12457878142595291 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09929823875427246 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.22963090240955353 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13553598523139954 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14491117000579834 *jet/accuracy_2_of_4‚0.6984360814094543 'particle/f_score‚0.6161037087440491 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5015044212341309 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.776001513004303 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21359527111053467 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6342761516571045 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9971033930778503 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39905256032943726 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6368244886398315 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940407276153564 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6891058683395386 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚263 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06192578002810478 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908832311630249 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927964806556702 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18377842009067535 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24963362514972687 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5338799953460693 9J0Xy"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5282528400421143 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408333539962769 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9881417155265808 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5209841132164001 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6595772504806519 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49678874015808105 /particle/accuracy_3_of_4‚0.9910430908203125 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17321521043777466 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6805555820465088 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04109556972980499 particle/accuracy_1_of_1‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7883597612380981 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5844992399215698 (particle/accuracy‚0.8396035432815552 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15904419124126434 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21417778730392456 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9945461750030518 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.693157434463501 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5069444179534912 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.44638341665267944 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 +jet/accuracy_2_of_2‚0.11820273101329803 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28673845529556274 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45208507776260376 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 )jet/accuracy_1_of_1‚0.256847620010376 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6572683453559875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1422572135925293 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4620259702205658 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1664140820503235 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5087435245513916 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6133448481559753 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22300389409065247 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2682737708091736 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5834325551986694 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5817684531211853 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020837614312767982 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17885252833366394 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19976241886615753 *jet/accuracy_1_of_3‚0.8885303735733032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5332543253898621 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19104668498039246 +particle/sensitivity‚0.5993773937225342 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8507934808731079 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891124963760376 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.0140206590294837957Ï×ð>šú{ 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1664140820503235 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5087435245513916 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6133448481559753 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22300389409065247 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2682737708091736 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5834325551986694 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5817684531211853 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020837614312767982 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17885252833366394 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19976241886615753 *jet/accuracy_1_of_3‚0.8885303735733032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5332543253898621 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19104668498039246 +particle/sensitivity‚0.5993773937225342 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8507934808731079 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891124963760376 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.014020659029483795 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5848214030265808 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_4_of_4‚0.9340416193008423 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5350735187530518 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5857071876525879 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6346451640129089 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17026135325431824 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5890712141990662 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7250980734825134 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957764148712158 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8905787467956543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4894252121448517 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6097177863121033 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8628289103507996 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18706917762756348 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5706607103347778 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5738416314125061 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5418106913566589 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20864075422286987 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.29158318042755127 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994839072227478 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9228695631027222 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7657595872879028 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14020821452140808 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚7 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7828420996665955 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3199 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02775241993367672 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933775663375854 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48920994997024536 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14491117000579834 +jet/accuracy_3_of_4‚0.08203098177909851 +particle/specificity‚0.9052958488464355 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5777212977409363 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5794214606285095 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5867123603820801 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941057562828064 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19764435291290283 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_1_of_5‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18072277307510376 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2516794800758362 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5383405685424805 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23400159180164337 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.235098198056221 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959599375724792 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953079223632812 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16502726078033447 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02275569923222065 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1804264783859253 *jet/accuracy_1_of_2‚0.7212826609611511 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48920994997024536 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03192296624183655 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20529361069202423 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950741529464722 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7196440100669861 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7001133561134338 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6891058683395386 particle/accuracy_4_of_5‚1 % _timestamp‚1.7843250681092362e+09 )loss/val_total_loss‚9.356665875195947 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16502726078033447 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17767715454101562 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.643361508846283 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19927656650543213 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6805555820465088 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7validation_average_jet_accuracy‚0.30567559599876404 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5305968523025513 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.9477324485778809 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7865269184112549 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9938897490501404 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10120194405317307 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_5_of_5‚0 _runtime‚5321.484594143 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953079223632812 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14273110032081604 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49780377745628357 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6341850161552429 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4957408308982849 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5871070623397827 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14099855720996857 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12457878142595291 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09929823875427246 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.22963090240955353 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13553598523139954 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14491117000579834 *jet/accuracy_2_of_4‚0.6984360814094543 'particle/f_score‚0.6161037087440491 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5015044212341309 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.776001513004303 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21359527111053467 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6342761516571045 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9971033930778503 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39905256032943726 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6368244886398315 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940407276153564 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6891058683395386 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚263 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06192578002810478 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908832311630249 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927964806556702 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18377842009067535 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24963362514972687 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5338799953460693 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5282528400421143 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408333539962769 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9881417155265808 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5209841132164001 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6595772504806519 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49678874015808105 /particle/accuracy_3_of_4‚0.9910430908203125 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17321521043777466 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6805555820465088 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04109556972980499 particle/accuracy_1_of_1‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7883597612380981 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5844992399215698 (particle/accuracy‚0.8396035432815552 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15904419124126434 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21417778730392456 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9945461750030518 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.693157434463501 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5069444179534912 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.44638341665267944 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 +jet/accuracy_2_of_2‚0.11820273101329803 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28673845529556274 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45208507776260376 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 )jet/accuracy_1_of_1‚0.256847620010376 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6572683453559875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1422572135925293 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4620259702205658‡§=‚Ÿ›‚(:” ÏÇêÒ×Ø”¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p¢ÌD5œ‚(:ª ÏÇêÒŸÌà®) proc.memory.percent‚17.50483512878418 disk.sda.in‚ 0.0703125 disk.sda.out‚ 349.38671875 network.recv‚ 1.364176e+06 disk.dm-1.out‚ 349.38671875$ disk.nvme0c0n1.in‚10352.64453125) disk./.usagePercent‚39.45865220266425 disk.drbd2165.in‚0& disk.nvme0n1.out‚61373.00439453125 disk.rbd1.in‚ 506.4921875" disk.rbd1.out‚17766.4248046875# memory_percent‚88.2986068725586$ disk./.usageGB‚587.9624328613281 disk.rbd2.out‚ 34.86328125( disk.nvme0c0n1.out‚61373.00439453125 disk.sda2.in‚0" proc.memory.rssMB‚ 2867.9921875 cpu‚25.495320648721783 network.sent‚ 5.972823e+06 disk.rbd2.in‚0.015625 disk.dm-0.in‚10352.64453125# disk.dm-0.out‚61373.00439453125 disk.sda1.out‚0 disk.sda3.out‚ 349.38671875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 1917.15625" disk.nvme0n1.in‚10352.64453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125#Ñ£…ž‚(:“ ÞÇêÒ›Ÿê­ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;(~R%ž‚(:š ÞÇêҙΦ¯ disk.sda3.in‚ 0.0703125 disk.nvme0n1.in‚ 10417.28125 disk.sda.out‚ 349.94140625" disk.rbd1.out‚17766.5498046875) proc.memory.percent‚17.50502586364746 disk.dm-1.out‚ 349.94140625% disk.nvme0c0n1.out‚61376.47265625 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.0703125# memory_percent‚88.2969856262207 cpu‚25.49593406836848# disk./.usageGB‚587.962272644043 disk.rbd2.out‚34.96923828125 proc.cpu.threads‚17) disk./.usagePercent‚39.45864145034781# disk.nvme0n1.out‚61376.47265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚61376.47265625 network.sent‚ 6.029661e+06 disk.dm-0.in‚ 10417.28125 disk.sda.in‚ 0.0703125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1917.421875" proc.memory.rssMB‚ 2868.0234375 disk.sda3.out‚ 349.94140625! disk.nvme0c0n1.in‚ 10417.28125 network.recv‚ 1.368174e+06 disk.rbd1.in‚ 506.4921875 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0P}H)”Ÿ‚(:‰ íÇêÒ‰†Ñ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿzi$- ‚(:¢ íÇêҺ檮 network.recv‚ 1.371936e+06 disk.dm-0.in‚10418.26171875 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚88.29700946807861 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.out‚350.375' disk.nvme0c0n1.out‚61387.1142578125 disk.sda.out‚350.375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2868.05078125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0703125) proc.memory.availableMB‚ 1917.41796875$ disk./.usageGB‚587.9622802734375" disk.dm-0.out‚61387.1142578125 disk.dm-1.out‚350.375* proc.memory.percent‚17.505192756652832 cpu‚25.496605616353385 network.sent‚ 6.034771e+06$ disk.nvme0c0n1.in‚10418.26171875% disk.nvme0n1.out‚61387.1142578125" disk.rbd1.out‚17766.5498046875" disk.nvme0n1.in‚10418.26171875 disk.sda.in‚ 0.0703125 disk.rbd1.in‚ 506.4921875 disk.rbd2.in‚0.015625 disk.rbd2.out‚34.96923828125 proc.cpu.threads‚17* disk./.usagePercent‚39.458641962362876 disk.sda2.out‚0 disk.rbd0.out‚0¹Ó“.¡‚(:£ üÇêÒåúÜ® disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 35.0400390625 disk.sda1.in‚0 disk.nvme0n1.in‚ 10419.09375( proc.memory.availableMB‚ 1917.1640625 cpu‚25.497213562477725 disk.rbd0.in‚0 disk.sda.in‚ 0.0703125 disk.dm-1.out‚ 351.421875 disk.sda2.out‚0 disk.sda3.in‚ 0.0703125 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 6.039822e+06# disk.dm-0.out‚61395.89208984375( disk.nvme0c0n1.out‚61395.89208984375 disk.drbd2165.in‚0& disk.nvme0n1.out‚61395.89208984375$ memory_percent‚88.29855918884277# proc.memory.rssMB‚ 2868.05078125 network.recv‚ 1.375273e+06* disk./.usagePercent‚39.458450980742334 disk.dm-0.in‚ 10419.09375 disk.dm-1.in‚ 0.0703125" disk.rbd1.out‚17766.6123046875$ disk./.usageGB‚587.9594345092773 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.out‚ 351.421875! disk.nvme0c0n1.in‚ 10419.09375 disk.sda.out‚ 351.421875 disk.rbd1.in‚ 506.4921875* proc.memory.percent‚17.505192756652832=²_¢‚(:’ üÇêÒ¬äçÙ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 7%ž£‚(:“ ‹ÈêÒõ–å« gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚254.229' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.84580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üFˆn3¤‚(:¨ ‹ÈêÒœÖÉ® disk.sda3.out‚ 351.76953125( disk.nvme0c0n1.out‚61402.68701171875 disk.dm-0.in‚ 10419.09375 disk.sda1.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2868.05078125* proc.memory.percent‚17.505192756652832 network.recv‚ 1.378784e+06 disk.sda.in‚ 0.0703125 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚88.30008506774902& disk.nvme0n1.out‚61402.68701171875" disk.rbd1.out‚17766.6123046875 disk.sda1.out‚0 disk.dm-1.out‚ 351.76953125! disk.nvme0c0n1.in‚ 10419.09375 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0703125 disk.nvme0n1.in‚ 10419.09375 disk.rbd1.in‚ 506.4921875 disk.rbd2.in‚0.015625# disk.dm-0.out‚61402.68701171875( proc.memory.availableMB‚ 1916.9140625 network.sent‚ 6.044912e+06 disk.dm-1.in‚ 0.0703125 disk.sda2.in‚0 disk.rbd2.out‚ 35.2666015625 cpu‚25.498054609625985) disk./.usagePercent‚39.45844458055398$ disk./.usageGB‚587.9593391418457 disk.sda.out‚ 351.76953125 disk.rbd0.in‚0 proc.cpu.threads‚17°!Ξ¥‚(:“ šÈêÒ‚û¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.855999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂYÿ(>¦‚(:³ šÈêÒ…§ ® proc.cpu.threads‚17$ disk./.usageGB‚587.9586067199707 disk.dm-1.in‚ 0.0703125 disk.sda2.in‚0# proc.memory.rssMB‚ 2868.05078125 disk.rbd1.in‚ 506.4921875) proc.memory.availableMB‚ 1917.22265625& disk.nvme0n1.out‚61411.43603515625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625$ disk.nvme0c0n1.in‚10420.05078125( disk.nvme0c0n1.out‚61411.43603515625 disk.sda2.out‚0$ memory_percent‚88.29820156097412 network.recv‚ 1.382277e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 352.24609375 disk.sda3.in‚ 0.0703125" disk.nvme0n1.in‚10420.05078125* proc.memory.percent‚17.505192756652832" disk.rbd1.out‚17766.6748046875 disk.dm-0.in‚10420.05078125# disk.dm-0.out‚61411.43603515625 disk.sda1.out‚0 disk.sda.in‚ 0.0703125 cpu‚25.498708356429283 network.sent‚ 6.050515e+06 disk.drbd2165.out‚0* disk./.usagePercent‚39.458395427107405 disk.rbd2.out‚ 35.3447265625 disk.sda3.out‚ 352.24609375 disk.sda.out‚ 352.24609375?] (2§- _wandbruntime‚5400  _runtime‚5400-«¢¿Ç¨Á & lr-AdamW/pg1‚0.0002600250830584707 & lr-AdamW/pg2‚0.0002600250830584707 trainer/global_step‚3249 % _timestamp‚1.7843251471430342e+09 _runtime‚5400.517693066 _step‚2643ÆpeÌ©Æ & lr-AdamW/pg1‚0.0002600250830584707 & lr-AdamW/pg2‚0.0002600250830584707 trainer/global_step‚3249 % _timestamp‚1.7843251471430342e+09 _runtime‚5400.517693066 _step‚264ˆ½©lV2ª- _wandbruntime‚5400  _runtime‚5400ƒ 3 ;«éO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3052684783355 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28152014 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256351758390784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937270271928320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3056571516928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256351355763200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚5400.519008239 \HDeviceStatsMonito9Ý0´r.on_train_batch_start/requested_bytes.large_pool.freed‚933809538936832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9242055 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27729919 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8340298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28142041 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚936862249148155 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933809564364800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252407474294784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943881468416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27729913 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934213726584832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3057916906496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881933 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8340293 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3051340183033 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27729919 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚3249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943916004864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937270271928320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28152014 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937271643491328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3057916906496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934213700411392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚936860879119865 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚3249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943916004864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937270271928320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28152014 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937271643491328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3057916906496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934213700411392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚936860879119865 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17587679 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888638 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934087446142976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934491701141504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3945120418304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3052263649279 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898631 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934491701141504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚937139684364287 _runtime‚5401.915691299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28160506 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27738119 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3058841949184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9244843 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937549197693952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28150519 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚937141054392689 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28150519 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28160506 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8342841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17587928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252484960702464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053608249713 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256430449211904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚266 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚934087420715008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058841949184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚937550569264128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3057496552448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3057496552448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937550569264128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252485328793600 % _timestamp‚1.7843251485408084e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934491727314944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934491727314944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898631 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888638 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8342836 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9245087 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28160506 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8342841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17587928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252484960702464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053608249713 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256430449211904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚266 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚934087420715008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058841949184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚937550569264128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3057496552448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3057496552448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937550569264128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252485328793600 % _timestamp‚1.7843251485408084e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934491727314944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934491727314944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898631 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_byt¸•wXôes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888638 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8342836 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9245087 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17587679 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888638 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934087446142976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934491701141504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3945120418304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3052263649279 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898631 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934491701141504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚937139684364287 _runtime‚5401.915691299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28160506 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27738119 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3058841949184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9244843 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937549197693952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28150519 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚937141054392689 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28150519 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256281794678784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950802575170063 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28567793 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚951217995927552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951216624364544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948113736916992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8467448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚951216624364544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28577766 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17848434 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚948113736916992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28577766 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28140213 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28567793 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚947703653273600 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28140219 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948113763090432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3102887447552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4003504739840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951217995927552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948113763090432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9381265 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256282162769920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3104232837120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3104232837120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3102887447552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260285704143360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717985 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28140219 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¬RÏÜPÂíO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28140219 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948113763090432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3102887447552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4003504739840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951217995927552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948113763090432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9381265 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256282162769920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3104232837120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3104232837120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3102887447552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260285704143360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717985 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28140219 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256281794678784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950802575170063 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28567793 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚951217995927552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951216624364544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948113736916992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8467448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚951216624364544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28577766 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17848434 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚948113736916992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28577766 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28140213 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28567793 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚947703653273600 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979âºZT2Ã- _wandbruntime‚5482  _runtime‚5482á¶T¸ù&ÄóM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951081380414597 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚5482.109761597 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28148419 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28586258 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3103812483072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724690 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3103812483072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8469991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28576271 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚947981509623808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17853994 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4004725396992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948391737647104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4004688770560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951496921700352 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951496921700352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3105157879808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256359281086464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3098500762387 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8469986 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843252287348845e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948391763820544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚270 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951495550130176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3105157879808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734683 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256359649177600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948391763820544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28586258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17853724 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948391737647104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28148419 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28576271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260363969857024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9383738 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734683 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820<7YÌþ&ÅøM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951496921700352 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951496921700352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3105157879808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256359281086464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3098500762387 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8469986 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843252287348845e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948391763820544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚270 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951495550130176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3105157879808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734683 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256359649177600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948391763820544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28586258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17853724 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948391737647104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28148419 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28576271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260363969857024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9383738 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734683 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951081380414597 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚5482.109761597 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28148419 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28586258 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3103812483072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724690 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3103812483072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8469991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28576271 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚947981509623808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17853994 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4004725396992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948391737647104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4004688770560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Ž~#ñ‹2Æ- _wandbruntime‚5482  _runtime‚5482tZ1Çþ 2loss/SRqqt2/assignment_loss‚0.6348191284497255 1loss/SRqqt1/detection_loss‚0.6444260238692967 1loss/SRqqt2/detection_loss‚0.9072770559204286 _runtime‚5482.190524625 0loss/FRt2/assignment_loss‚1.8607502097256796 .loss/FRt1/detection_loss‚1.446115737151302 $ _timestamp‚1.784325228816076e+09 %loss/total_loss‚9.146572482218193 epoch‚8 0loss/FRt1/assignment_loss‚0.7427056129549591 1loss/FBt1/assignment_loss‚0.05284570213468921 0loss/FBt2/assignment_loss‚0.0846257806934177 /loss/FRt2/detection_loss‚0.5983962213021156 trainer/global_step‚3299 _step‚271 /loss/FBt1/detection_loss‚0.5532145561865851 /loss/FBt2/detection_loss‚1.0857853311403045 2loss/SRqqt1/assignment_loss‚0.5356111226896905 …WÛ Èƒ /loss/FBt1/detection_loss‚0.5532145561865851 /loss/FBt2/detection_loss‚1.0857853311403045 2loss/SRqqt1/assignment_loss‚0.5356111226896905 2loss/SRqqt2/assignment_loss‚0.6348191284497255 1loss/SRqqt1/detection_loss‚0.6444260238692967 1loss/SRqqt2/detection_loss‚0.9072770559204286 _runtime‚5482.190524625 0loss/FRt2/assignment_loss‚1.8607502097256796 .loss/FRt1/detection_loss‚1.446115737151302 $ _timestamp‚1.784325228816076e+09 %loss/total_loss‚9.146572482218193 epoch‚8 0loss/FRt1/assignment_loss‚0.7427056129549591 1loss/FBt1/assignment_loss‚0.05284570213468921 0loss/FBt2/assignment_loss‚0.0846257806934177 /loss/FRt2/detection_loss‚0.5983962213021156 trainer/global_step‚3299 _step‚271|ÚÓŸÉ‚(:” ôÈêҘ癬 gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ̽Ê‚(:ˆ ôÈêÒÒú©® disk.dm-1.out‚354% disk.nvme0c0n1.out‚61458.10546875 disk.sda2.in‚0 network.sent‚ 6.138733e+06) disk./.usagePercent‚39.45847146134508 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 10548.640625 cpu‚25.502264631637257 disk.sda.in‚ 0.2421875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚ 0.2421875 disk.sda3.in‚ 0.2421875 disk.sda.out‚354 disk.rbd1.in‚ 506.4921875" disk.rbd1.out‚17767.0537109375$ memory_percent‚88.30130100250244 disk.dm-0.in‚ 10548.640625 disk.nvme0n1.in‚ 10548.640625* proc.memory.percent‚17.505192756652832 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 36.2861328125 disk.drbd2165.in‚0# disk.nvme0n1.out‚61458.10546875 disk.sda2.out‚0$ disk./.usageGB‚587.9597396850586# proc.memory.rssMB‚ 2868.05078125 network.recv‚ 1.404574e+06 disk.sda1.out‚0 disk.sda3.out‚354) proc.memory.availableMB‚ 1916.71484375 disk.rbd2.in‚0.015625 disk.dm-0.out‚61458.10546875¦âíŸË‚(:” ƒÉêÒ¯²¾¬ gpu.0.gpu‚77.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïr½"5Ì‚(:ª ƒÉêҺܦ¯ disk.sda1.in‚0$ disk./.usageGB‚587.9598007202148 disk.rbd0.out‚0 disk.dm-0.in‚ 10548.640625( disk.nvme0c0n1.out‚61465.74755859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚ 6.20182e+06) disk./.usagePercent‚39.45847555746562$ memory_percent‚88.30130100250244 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda3.out‚ 354.12109375 disk.sda.in‚ 0.2421875 disk.sda.out‚ 354.12109375 cpu‚25.50270124840699 disk.dm-1.out‚ 354.12109375# proc.memory.rssMB‚ 2868.05078125* proc.memory.percent‚17.505192756652832" disk.rbd1.out‚17767.0537109375 network.recv‚ 1.409963e+06 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.2421875" disk.nvme0c0n1.in‚ 10548.640625 disk.nvme0n1.in‚ 10548.640625& disk.nvme0n1.out‚61465.74755859375 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 36.2861328125# disk.dm-0.out‚61465.74755859375 disk.rbd1.in‚ 506.4921875 disk.dm-1.in‚ 0.2421875) proc.memory.availableMB‚ 1916.71484375º ÖøžÍ‚(:“ ’ÉêÒ´…Ϋ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.89299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\>(ø2΂(:§ ’ÉêÒ‚”Ê­( disk.nvme0c0n1.out‚61474.32861328125" disk.nvme0n1.in‚10549.47265625 disk.sda.in‚ 0.2421875 disk.drbd2165.in‚0& proc.memory.availableMB‚ 1916.71875# proc.memory.rssMB‚ 2868.05078125 network.sent‚ 6.206858e+06 disk.sda.out‚ 354.484375 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚88.30127716064453! disk./.usageGB‚587.9599609375 disk.sda3.in‚ 0.2421875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚10549.47265625 disk.drbd2165.out‚0* proc.memory.percent‚17.505192756652832$ disk.nvme0c0n1.in‚10549.47265625 disk.rbd2.out‚36.34912109375 cpu‚25.50343186663547 proc.cpu.threads‚17) disk./.usagePercent‚39.45848630978206 disk.sda3.out‚ 354.484375 network.recv‚ 1.413403e+06& disk.nvme0n1.out‚61474.32861328125 disk.sda2.out‚0# disk.rbd1.out‚17767.48974609375 disk.sda1.out‚0 disk.rbd1.in‚ 506.4921875# disk.dm-0.out‚61474.32861328125 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 354.484375ýDžÏ‚(:“ ¡ÉêÒÒÅè« gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.54859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€pÇU6Ђ(:« ¡ÉêÒâ¹Ç® network.sent‚ 6.212072e+06$ disk./.usageGB‚587.9598083496094 disk.dm-0.in‚10610.30078125 disk.drbd2165.out‚0 disk.sda3.out‚ 354.734375" disk.nvme0n1.in‚10610.30078125$ disk.nvme0c0n1.in‚10610.30078125( disk.nvme0c0n1.out‚61486.37255859375 disk.sda.out‚ 354.734375 disk.sda2.in‚0& proc.memory.availableMB‚ 1916.71875 disk.sda1.in‚0 disk.sda.in‚ 0.2421875# disk.rbd1.out‚17767.48974609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 354.734375 disk.sda3.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875 cpu‚25.503952768749684 proc.cpu.threads‚17) disk./.usagePercent‚39.45847606948069 disk.rbd2.in‚0.015625 disk.rbd2.out‚36.34912109375# disk.dm-0.out‚61486.37255859375* proc.memory.percent‚17.505192756652832 network.recv‚ 1.417047e+06 disk.dm-1.in‚ 0.2421875 disk.sda2.out‚0$ memory_percent‚88.30127716064453# proc.memory.rssMB‚ 2868.05078125 disk.drbd2165.in‚0& disk.nvme0n1.out‚61486.37255859375¿‚IÑÑ‚(:’ °ÉêÒŒú­ gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÊÜ 1Ò‚(:¦ °ÉêҘм® disk.rbd2.out‚ 37.1357421875# proc.memory.rssMB‚ 2868.05078125% disk.nvme0c0n1.out‚61490.61328125 disk.sda.out‚ 355.76171875 disk.sda1.in‚0& proc.memory.availableMB‚ 1916.71875$ disk./.usageGB‚587.9598770141602 disk.sda3.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875 network.recv‚ 1.420553e+06 disk.dm-0.in‚10610.30078125 disk.dm-0.out‚61490.61328125 cpu‚25.504660480736547 network.sent‚ 6.21768e+06$ disk.nvme0c0n1.in‚10610.30078125" disk.nvme0n1.in‚10610.30078125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 355.76171875) disk./.usagePercent‚39.45848067761631$ memory_percent‚88.30127716064453 proc.cpu.threads‚17# disk.rbd1.out‚17767.55224609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.2421875 disk.sda.in‚ 0.2421875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625* proc.memory.percent‚17.505192756652832 disk.sda3.out‚ 355.76171875# disk.nvme0n1.out‚61490.61328125&Uã¡2Ó- _wandbruntime‚5561  _runtime‚5561|ïÍÅÔ¿ & lr-AdamW/pg1‚0.0002590408126182775 & lr-AdamW/pg2‚0.0002590408126182775 trainer/global_step‚3349 $ _timestamp‚1.784325307765067e+09 _runtime‚ 5561.13967231 _step‚272kOrÊÕÄ _step‚272 & lr-AdamW/pg1‚0.0002590408126182775 & lr-AdamW/pg2‚0.0002590408126182775 trainer/global_step‚3349 $ _timestamp‚1.784325307765067e+09 _runtime‚ 5561.13967231ºÌH~2Ö- _wandbruntime‚5561  _runtime‚5561÷`ñIp×èO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatÚ@Ü~'sMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28993545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4062828094976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚964741531163681 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28550519 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚962013773422592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚5561.140961264 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264218943157760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961597716754432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3149203378176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965162976800768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚962013773422592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962013799596032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3150548767744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3149203378176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 _step‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8594598 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4062864728576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3145159009571 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28550513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚965162976800768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3143814409249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚965164348363776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29003518 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9518789 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260156115062784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260156483153920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264219347882496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28550519 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962013799596032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚964742901191971 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 $ _timestamp‚1.784325307766183e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600Œ×ó'ØíO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264218943157760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961597716754432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3149203378176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965162976800768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚962013773422592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962013799596032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3150548767744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3149203378176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 _step‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8594598 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4062864728576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3145159009571 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28550513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚965162976800768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3143814409249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚965164348363776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29003518 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9518789 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260156115062784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260156483153920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264219347882496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28550519 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962013799596032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚964742901191971 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 $ _timestamp‚1.784325307766183e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28993545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4062828094976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚964741531163681 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28550519 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚962013773422592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚5561.140961264 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962291774152704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3151473810432 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8597136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4064046272000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28558725 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29012010 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962291800326144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260233969561600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚965443274136576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962291800326144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚274 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560742 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18118679 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3151473810432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚965443274136576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28558725 trainer/global_step‚3349 _runtime‚5562.538502382 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4064009645568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3146082475929 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965020336408103 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150128413696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3144737875495 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29002023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260233601470464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965441902566400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264298015833600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570735 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28558719 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3150128413696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18118945 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8597141 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264297611116032ŽÔS²A ÛøM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29002023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260233601470464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965441902566400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264298015833600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570735 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28558719 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.€{½on_train_batch_end/active_bytes.small_pool.freed‚ 3150128413696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18118945 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8597141 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264297611116032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962291774152704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3151473810432 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8597136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4064046272000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28558725 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29012010 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962291800326144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260233969561600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚965443274136576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962291800326144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚274 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560742 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18118679 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3151473810432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚965443274136576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28558725 trainer/global_step‚3349 _runtime‚5562.538502382 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4064009645568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3146082475929 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965020336408103 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150128413696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3144737875495 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008’³åÎ,2Ü- _wandbruntime‚5562  _runtime‚55622§özÝÿ 1loss/FBt2/assignment_loss‚0.04228671350489128 1loss/SRqqt2/detection_loss‚1.0488254805377712 0loss/FRt1/assignment_loss‚0.8767975191317642 3loss/SRqqt1/assignment_loss‚0.47679489229419736 2loss/SRqqt2/assignment_loss‚0.6122477119206451 /loss/FBt1/detection_loss‚0.5563136195821298 %loss/total_loss‚9.141112204408643 epoch‚8 trainer/global_step‚3349 % _timestamp‚1.7843253092439802e+09 /loss/FRt2/assignment_loss‚1.924644879084747 2loss/FBt1/assignment_loss‚0.040292404205746465 1loss/SRqqt1/detection_loss‚0.7052650013492116 .loss/FBt2/detection_loss‚0.765271390816479 _runtime‚ 5562.61858734 .loss/FRt1/detection_loss‚1.487501445228357 /loss/FRt2/detection_loss‚0.6048711467527014 _step‚275h$$! Þ„ epoch‚8 trainer/global_step‚3349 % _timestamp‚1.7843253092439802e+09 /loss/FRt2/assignment_loss‚1.924644879084747 2loss/FBt1/assignment_loss‚0.040292404205746465 1loss/SRqqt1/detection_loss‚0.7052650013492116 .loss/FBt2/detection_loss‚0.765271390816479 _runtime‚ 5562.61858734 .loss/FRt1/detection_loss‚1.487501445228357 /loss/FRt2/detection_loss‚0.6048711467527014 _step‚275 1loss/FBt2/assignment_loss‚0.04228671350489128 1loss/SRqqt2/detection_loss‚1.0488254805377712 0loss/FRt1/assignment_loss‚0.8767975191317642 3loss/SRqqt1/assignment_loss‚0.47679489229419736 2loss/SRqqt2/assignment_loss‚0.6122477119206451 /loss/FBt1/detection_loss‚0.5563136195821298 %loss/total_loss‚9.141112204408643“fª'”ß‚(:‰ ¿ÉêÒ’™Ì¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚331.497' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÇ4à‚(:© ¿ÉêÒ»Á¦®# memory_percent‚88.2955551147461 disk.sda1.out‚0 disk.rbd1.in‚ 506.4921875 disk.dm-0.in‚ 10611.1328125# proc.memory.rssMB‚ 2868.05078125 network.sent‚ 6.222249e+06 disk.sda3.out‚ 356.41796875# disk.nvme0c0n1.in‚ 10611.1328125 disk.rbd0.in‚0 disk.sda.in‚ 0.2421875 disk.dm-1.in‚ 0.2421875* proc.memory.percent‚17.505192756652832 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 356.41796875& proc.memory.availableMB‚ 1917.65625 disk.sda1.in‚0$ disk./.usageGB‚587.9597434997559' disk.nvme0c0n1.out‚61502.9873046875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 37.1357421875) disk./.usagePercent‚39.45847171735261! disk.nvme0n1.in‚ 10611.1328125% disk.nvme0n1.out‚61502.9873046875# disk.rbd1.out‚17767.55224609375 cpu‚25.505186329583342 network.recv‚ 1.423971e+06 disk.sda3.in‚ 0.2421875" disk.dm-0.out‚61502.9873046875 proc.cpu.threads‚17 disk.sda.out‚ 356.41796875Ïì”á‚(:‰ ÎÉêÒß®¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö•U#2â‚(:§ ÎÉêÒâÂð® disk.rbd0.in‚0 disk.rbd2.out‚37.17919921875 disk.dm-0.out‚61511.01171875# proc.memory.rssMB‚ 2868.05078125 proc.cpu.threads‚17 network.recv‚ 1.428814e+06) disk./.usagePercent‚39.45847325339781# disk.rbd1.out‚17767.61474609375( proc.memory.availableMB‚ 1918.0703125 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875 disk.rbd2.in‚0.015625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚10612.08984375% disk.nvme0c0n1.out‚61511.01171875 network.sent‚ 6.285533e+06 disk.drbd2165.out‚0 disk.dm-1.out‚ 357.0234375 disk.sda.in‚ 0.2421875 cpu‚25.505612992799264 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.2421875 disk.sda.out‚ 357.0234375* proc.memory.percent‚17.505192756652832 disk.dm-0.in‚10612.08984375 disk.dm-1.in‚ 0.2421875 disk.sda3.out‚ 357.0234375" disk.nvme0n1.in‚10612.08984375 disk.drbd2165.in‚0$ disk./.usageGB‚587.9597663879395 disk.rbd0.out‚0# disk.nvme0n1.out‚61511.01171875$ memory_percent‚88.29302787780762./’”ã‚(:‰ ÝÉêÒà›¸® gpu.0.gpu‚49.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¥ü$4ä‚(:© ÝÉêÒŽØ­¯ disk.dm-0.out‚61519.33203125$ disk./.usageGB‚587.9597702026367 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚10669.96484375 disk.sda.out‚ 357.36328125$ memory_percent‚88.29450607299805 disk.rbd2.in‚0.015625 disk.rbd2.out‚37.17919921875 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.out‚61519.33203125 disk.rbd0.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1917.828125 cpu‚25.506305297248854 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚10669.96484375 disk.sda3.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875 disk.dm-1.out‚ 357.36328125 disk.sda.in‚ 0.2421875# disk.rbd1.out‚17767.61474609375% disk.nvme0c0n1.out‚61519.33203125" disk.nvme0n1.in‚10669.96484375 network.sent‚ 6.290633e+06 network.recv‚ 1.432404e+06) disk./.usagePercent‚39.45847350940535 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.2421875# proc.memory.rssMB‚ 2868.05078125 disk.sda3.out‚ 357.36328125* proc.memory.percent‚17.505192756652832-ÈMŸå‚(:” ìÉêÒñÃã« gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.480399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ›+æ‚(:  ìÉêÒ¦ê­ network.recv‚ 1.435682e+06# disk.rbd1.out‚17767.67724609375 disk.dm-1.out‚ 358.23828125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.2421875$ disk./.usageGB‚587.9578437805176% disk.nvme0c0n1.out‚61528.96484375 disk.nvme0n1.in‚ 10670.796875 disk.sda2.out‚0' proc.memory.availableMB‚ 1917.828125 disk.sda3.out‚ 358.23828125 disk.rbd2.in‚0.015625 disk.dm-0.out‚61528.96484375" disk.nvme0c0n1.in‚ 10670.796875# disk.nvme0n1.out‚61528.96484375 disk.sda.in‚ 0.2421875 disk.rbd0.out‚0 disk.drbd2165.in‚0 cpu‚25.506879674539807 network.sent‚ 6.295687e+06* disk./.usagePercent‚39.458344225600555# proc.memory.rssMB‚ 2868.05078125 disk.sda.out‚ 358.23828125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 37.2421875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚88.29450607299805 disk.sda3.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875 disk.dm-0.in‚ 10670.796875* proc.memory.percent‚17.505192756652832Û1n.”ç‚(:‰ ûÉêÒŸ€¾¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ŠY.7è‚(:¬ ûÉêÒ†Ýø® network.recv‚ 1.439442e+06$ disk./.usageGB‚587.9578742980957 disk.sda2.out‚0" disk.dm-0.out‚61541.7099609375 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1918.07421875* proc.memory.percent‚17.505192756652832 network.sent‚ 6.301286e+06) disk./.usagePercent‚39.45834627366083# proc.memory.rssMB‚ 2868.05078125 cpu‚25.507426079733786 proc.cpu.threads‚17# disk.rbd1.out‚17767.67724609375 disk.rbd0.in‚0 disk.dm-0.in‚10671.75390625$ disk.nvme0c0n1.in‚10671.75390625 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.2421875 disk.sda3.out‚ 358.71484375' disk.nvme0c0n1.out‚61541.7099609375" disk.nvme0n1.in‚10671.75390625# memory_percent‚88.2930040359497 disk.sda.in‚ 0.2421875 disk.sda3.in‚ 0.2421875 disk.sda.out‚ 358.71484375 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0 disk.rbd2.out‚ 37.2421875 disk.sda1.out‚0 disk.dm-1.out‚ 358.71484375% disk.nvme0n1.out‚61541.7099609375 disk.rbd2.in‚0.015625>˜¸½žé‚(:“ ŠÊêÒ“¡º­ gpu.0.gpu‚84.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.618' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.72359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ýEÎ1ê‚(:¦ ŠÊêÒ˜Š·® network.recv‚ 1.442811e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 359.62109375 disk.sda1.in‚0 network.sent‚ 6.305945e+06 disk.sda2.in‚0# memory_percent‚88.2930040359497) proc.memory.availableMB‚ 1918.07421875$ disk./.usageGB‚587.9579544067383 disk.drbd2165.in‚0# disk.rbd1.out‚17767.80224609375 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 359.62109375 disk.sda3.in‚ 0.2421875" disk.nvme0n1.in‚10671.75390625 disk.sda.out‚ 359.62109375 disk.rbd2.in‚0.015625 cpu‚25.50807325858057 disk.sda2.out‚0 disk.rbd2.out‚37.30517578125 disk.dm-0.in‚10671.75390625) disk./.usagePercent‚39.45835164981905$ disk.nvme0c0n1.in‚10671.75390625 disk.sda.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875# proc.memory.rssMB‚ 2868.05078125* proc.memory.percent‚17.505192756652832$ disk.nvme0c0n1.out‚ 61545.6484375" disk.nvme0n1.out‚ 61545.6484375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.2421875 disk.dm-0.out‚ 61545.6484375*yŸ2ë- _wandbruntime‚5641  _runtime‚5641ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979109329236992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979110700800000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3196864698368 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28960819 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28960819 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264030435446784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191396122679 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975913809928192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390089 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚277 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268153089060352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264030803537920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚975491805663232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚975913836101632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28960813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8721743 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4122285522432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978683227213879 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18379017 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29429270 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9657269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4122250985984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚979109329236992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚1209755A$ÄðîO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚979109329236992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979109329236992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979110700800000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3196864698368 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28960819 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28960819 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264030435446784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191396122679 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975913809928192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390089 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚277 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268153089060352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264030803537920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚975491805663232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚975913836101632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ ²t028960813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8721743 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4122285522432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978683227213879 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18379017 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29429270 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9657269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4122250985984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175•lß@U2ñ- _wandbruntime‚5642  _runtime‚5642Êp¶Xø&òòM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396794 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28969025 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18384045 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4123475192320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4123440663040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29427775 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 trainer/global_step‚3399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396794 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8724286 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3197789741056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843253893511863e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3196444344320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268231362517504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9660007 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978960662430011 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3190974988603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28969019 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406787 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28969019 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264108289945600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352´±Y¹ý&ó÷M A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚278 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8724291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9659759 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3196444344320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268231362517504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9660007 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978960662430011 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3190974988603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28969019 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406787 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28969019 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264108289945600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396794 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28969025 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18384045 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4123475192320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4123440663040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29427775 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 trainer/global_step‚3399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396794 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8724286 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3197789741056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843253893511863e+09 Fø‚(:³ ™ÊêÒŸ²è® network.sent‚ 6.311105e+06 disk.dm-0.in‚10756.67578125# disk.rbd1.out‚17767.80224609375# proc.memory.rssMB‚ 2868.05078125$ disk./.usageGB‚587.9589576721191 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.2421875( disk.nvme0c0n1.out‚61563.00830078125* proc.memory.percent‚17.505192756652832* disk./.usagePercent‚39.458418979800555 disk.rbd2.out‚37.30517578125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.2421875" disk.nvme0n1.in‚10756.67578125 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0& disk.nvme0n1.out‚61563.00830078125 disk.sda2.in‚0# memory_percent‚88.2932186126709( proc.memory.availableMB‚ 1918.0390625 cpu‚25.508620520940074# disk.dm-0.out‚61563.00830078125 disk.dm-1.out‚ 360.40234375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda3.out‚ 360.40234375 disk.sda.in‚ 0.2421875 network.recv‚ 1.446472e+06$ disk.nvme0c0n1.in‚10756.67578125 disk.sda.out‚ 360.402343752!Z”ù‚(:‰ ¨ÊêÒ¿ÛÈ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%E=ú‚(:² ¨ÊêÒîŒÒ¯ disk.sda3.in‚ 0.2421875$ disk.nvme0c0n1.in‚10756.67578125( disk.nvme0c0n1.out‚61569.66552734375 disk.rbd0.in‚0 network.recv‚ 1.451387e+06 disk.sda1.out‚0 disk.dm-1.out‚ 362.66796875" disk.nvme0n1.in‚10756.67578125& disk.nvme0n1.out‚61569.66552734375 cpu‚25.50916092899823 disk.drbd2165.in‚0 disk.sda3.out‚ 362.66796875 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875# disk.rbd1.out‚17767.86474609375# proc.memory.rssMB‚ 2868.05078125* proc.memory.percent‚17.505192756652832 proc.cpu.threads‚17 network.sent‚ 6.374241e+06 disk.rbd0.out‚0( disk./.usagePercent‚39.4586301860163 disk.dm-1.in‚ 0.2421875 disk.sda.in‚ 0.2421875 disk.rbd2.in‚0.015625$ disk./.usageGB‚587.9621047973633 disk.sda1.in‚0 disk.sda.out‚ 362.66796875# disk.dm-0.out‚61569.66552734375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd2.out‚37.30517578125 disk.dm-0.in‚10756.67578125$ memory_percent‚88.29624652862549) proc.memory.availableMB‚ 1917.54296875‹Gñmžû‚(:“ ·ÊêÒ—Ѭ gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚328.402' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.68039999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½¸°k7ü‚(:¬ ·ÊêÒÒ–°$ disk./.usageGB‚587.9620895385742 disk.sda3.out‚ 363.4765625( disk.nvme0c0n1.out‚61581.51220703125* proc.memory.percent‚17.505192756652832 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚37.30517578125 disk.dm-1.in‚ 0.2421875) disk./.usagePercent‚39.45862916198617 disk.sda.in‚ 0.2421875 disk.dm-0.in‚ 10757.6328125 disk.sda1.in‚0 disk.sda.out‚ 363.4765625$ memory_percent‚88.29731941223145# proc.memory.rssMB‚ 2868.05078125 network.sent‚ 6.37914e+06# disk.nvme0c0n1.in‚ 10757.6328125 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 network.recv‚ 1.454464e+06 disk.sda2.in‚0# disk.rbd1.out‚17767.86474609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1917.3671875 cpu‚25.509920022290398 disk.sda3.in‚ 0.2421875! disk.nvme0n1.in‚ 10757.6328125& disk.nvme0n1.out‚61581.51220703125 disk.dm-1.out‚ 363.4765625 disk.rbd2.in‚0.015625# disk.dm-0.out‚61581.51220703125 disk.sda1.out‚0 proc.cpu.threads‚17ñ² ¾ˆý‚(:ˆ ÆÊêÒÉʳ¬ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocateN5 dBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.335' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.867 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹>”?þ‚(:´ ÆÊêÒú·Ä®# disk.rbd1.out‚17767.92724609375 disk.rbd2.out‚37.36767578125 disk.sda1.in‚0 disk.sda3.out‚ 363.78515625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚61589.25439453125" disk.nvme0n1.in‚10758.46484375 disk.sda.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875) proc.memory.availableMB‚ 1917.37109375# disk./.usageGB‚587.961368560791# disk.dm-0.out‚61589.25439453125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 6.383459e+06 network.recv‚ 1.457531e+06 disk.dm-0.in‚10758.46484375 disk.sda3.in‚ 0.2421875 cpu‚25.510389556402664 disk.sda2.out‚0 disk.rbd2.in‚0.015625$ memory_percent‚88.29729557037354* disk./.usagePercent‚39.458580776562194$ disk.nvme0c0n1.in‚10758.46484375 disk.sda.out‚ 363.78515625 disk.sda2.in‚0 disk.dm-1.in‚ 0.2421875& disk.nvme0n1.out‚61589.25439453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2868.05078125 disk.dm-1.out‚ 363.78515625* proc.memory.percent‚17.505192756652832c¡;Š”ÿ‚(:‰ ÕÊêÒ™³ö« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äeqO:€‚(:¯ ÕÊêÒ¶÷Ø­ disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.2421875* proc.memory.percent‚17.505192756652832$ disk./.usageGB‚587.9623336791992 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.rbd1.out‚17767.92724609375 proc.cpu.threads‚17 disk.dm-1.out‚ 364.17578125' disk.nvme0c0n1.out‚61595.0908203125 disk.sda.in‚ 0.2421875 disk.rbd0.in‚0 disk.rbd2.out‚37.36767578125 disk.dm-1.in‚ 0.2421875! disk.nvme0n1.in‚ 10841.4609375 disk.rbd1.in‚ 506.4921875# proc.memory.rssMB‚ 2868.05078125 network.sent‚ 6.388574e+06* disk./.usagePercent‚39.458645546468354 disk.sda3.out‚ 364.17578125% disk.nvme0n1.out‚61595.0908203125 disk.sda.out‚ 364.17578125 disk.sda2.in‚0 cpu‚25.510892844369824 network.recv‚ 1.461322e+06 disk.rbd0.out‚0 disk.sda1.out‚0" disk.dm-0.out‚61595.0908203125 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 10841.4609375$ memory_percent‚88.29843997955322) proc.memory.availableMB‚ 1917.18359375 disk.dm-0.in‚ 10841.4609375 disk.sda2.out‚0†š•2- _wandbruntime‚5721  _runtime‚5721gPˆÉ‚à ' lr-AdamW/pg1‚0.00025802519739876037 ' lr-AdamW/pg2‚0.00025802519739876037 trainer/global_step‚3449 % _timestamp‚1.7843254683954232e+09 _runtime‚5721.770068394 _step‚280uüïÎƒÈ _step‚280 ' lr-AdamW/pg1‚0.00025802519739876037 ' lr-AdamW/pg2‚0.00025802519739876037 trainer/global_step‚3449 % _timestamp‚1.7843254683954232e+09 _runtime‚5721.770068394˜ªJ2„- _wandbruntime‚5721  _runtime‚5721W&àï'…éO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226141 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989813846433792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3243180628992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚281 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚992623553235787 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29371113 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843254683965929e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18644372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216162 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993055681673216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚989813872607232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29845049 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4182386997760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848898 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8848893 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9795241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272087543358976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267904755830784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3241835239424 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9795474 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3241835239424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚993055681673216 trainer/global_step‚3449 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚993057053236224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29371119 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚989385920000000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29371113 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29855022 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3236288635465 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18644134 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚992622183207497 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29855022 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚5721.771378387 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216¼ øâô'†îO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216162 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993055681673216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚989813872607232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29845049 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4182386997760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848898 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8848893 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9795241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272087543358976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267904755830784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3241835239424 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9795474 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3241835239424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚993055681673216 trainer/global_step‚3449 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚993057053236224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29371119 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚989385920000000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29371113 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29855022 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3236288635465 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18644134 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚992622183207497 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29855022 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚5721.771378387 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226141 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989813846433792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3243180628992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚281 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚992623553235787 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29371113 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843254683965929e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18644372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0™Îcó$2‡- _wandbruntime‚5723  _runtime‚5723ÈB°¸„"ˆòM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990091847163904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242839 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29853527 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3244105671680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚993335979009024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232846 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18649379 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚990091847163904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272165828293120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9798166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29853527 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8851441 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚992900988451919 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4183618486784 trainer/global_step‚3449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29379319 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29379319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4183586054656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚993334607438848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚990091873337344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29863514 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267982242238464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232846 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29863514 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3242760274944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3244105671680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18649607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993335979009024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.0Ó†vtallocated‚272166228816384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8851436 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242839 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3238556702145 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9797943 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648Ûæ§Ìý&‰÷M TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3237212101711 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3242760274944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993335979009024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272166228816384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8851436 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242839 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3238556702145 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9797943 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990091847163904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242839 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29853527 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3244105671680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚993335979009024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232846 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18649379 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚990091847163904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272165828293120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9798166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29853527 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8851441 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚992900988451919 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4183618486784 trainer/global_step‚3449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29379319 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29379319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4183586054656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚993334607438848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚990091873337344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29863514 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267982242238464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232846 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29863514 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3242760274944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3244105671680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18649607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BŽ‚(:³ äÊêÒ¯­µ¯ disk.drbd2165.in‚0 disk.sda3.in‚ 0.2421875# disk.rbd1.out‚17767.98974609375 network.recv‚ 1.464627e+06 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 364.63671875$ disk.nvme0c0n1.in‚10842.41796875" disk.nvme0n1.in‚10842.41796875 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚10842.41796875$ disk./.usageGB‚587.9624519348145 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.2421875 cpu‚25.511630663067766 disk.dm-1.out‚ 364.63671875 network.sent‚ 6.394175e+06) disk./.usagePercent‚39.45865348270192& disk.nvme0n1.out‚61602.81201171875 disk.sda.in‚ 0.2421875 disk.rbd2.out‚ 37.3916015625# disk.dm-0.out‚61602.81201171875$ memory_percent‚88.30235004425049 disk.sda.out‚ 364.63671875) proc.memory.availableMB‚ 1916.54296875 disk.sda1.in‚0# proc.memory.rssMB‚ 2868.05078125( disk.nvme0c0n1.out‚61602.81201171875* proc.memory.percent‚17.505192756652832 proc.cpu.threads‚17)“‚(:ˆ óÊêÒ”û¬ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÖtw)‚(:ž óÊêÒ¨ûÒ®# disk.rbd1.out‚17767.98974609375 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚88.29958438873291* disk./.usagePercent‚39.458651946656715 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 365.05078125 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 365.05078125 disk.nvme0c0n1.in‚10843.25 disk.nvme0n1.in‚10843.25* proc.memory.percent‚17.505192756652832 disk.dm-0.in‚10843.25" disk.dm-0.out‚61613.0224609375 disk.sda3.in‚ 0.2421875' disk.nvme0c0n1.out‚61613.0224609375 network.sent‚ 6.457414e+06 disk.rbd1.in‚ 506.4921875 network.recv‚ 1.470119e+06 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.2421875# proc.memory.rssMB‚ 2868.05078125 proc.cpu.threads‚17$ disk./.usageGB‚587.9624290466309 disk.sda.in‚ 0.2421875 disk.rbd2.out‚ 37.3916015625% disk.nvme0n1.out‚61613.0224609375 cpu‚25.51208389190999 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1916.99609375 disk.sda.out‚ 365.05078125ª&²*Ÿ‘‚(:” ‚ËêÒ­ù« gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÚã],’‚(:¡ ‚ËêÒÐÐ¥®& disk.nvme0n1.out‚61615.96826171875$ disk./.usageGB‚587.9625244140625 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 365.54296875 proc.cpu.threads‚17# disk.dm-0.out‚61615.96826171875 disk.sda3.in‚ 0.2421875 disk.sda.out‚ 365.54296875* proc.memory.percent‚17.505192756652832 disk.dm-0.in‚10843.25 disk.sda1.out‚0( disk.nvme0c0n1.out‚61615.96826171875 disk.rbd1.in‚ 506.4921875 cpu‚25.512531222226873 network.recv‚ 1.473238e+06 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2868.05078125 disk.rbd0.in‚0 disk.sda3.out‚ 365.54296875) disk./.usagePercent‚39.45865834684507 disk.sda1.in‚0$ memory_percent‚88.30113410949707( proc.memory.availableMB‚ 1916.7421875 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚37.45458984375 disk.sda.in‚ 0.2421875 disk.sda2.in‚0 disk.sda2.out‚0# disk.rbd1.out‚17768.05224609375 network.sent‚ 6.461953e+06 disk.nvme0c0n1.in‚10843.25 disk.nvme0n1.in‚10843.25_ÅѺ”“‚(:‰ ‘ËêÒáü¯­ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïUÅ5”‚(:ª ‘ËêÒä ì® disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 6.467584e+06 disk.rbd0.in‚0 disk.rbd2.out‚37.45458984375! disk.dm-0.out‚61626.048828125 disk.dm-1.in‚ 0.2421875 proc.cpu.threads‚17# disk.rbd1.out‚17768.05224609375 disk.sda3.in‚ 0.2421875 disk.sda2.in‚0) proc.memory.availableMB‚ 1916.51171875 network.recv‚ 1.477039e+06! disk.nvme0n1.in‚ 10916.5546875 cpu‚25.51326966851433$ disk./.usageGB‚587.9623794555664 disk.sda3.out‚ 365.78515625& disk.nvme0c0n1.out‚61626.048828125$ disk.nvme0n1.out‚61626.048828125# proc.memory.rssMB‚ 2868.05078125 disk.rbd1.in‚ 506.4921875 disk.drbd2165.in‚0 disk.dm-1.out‚ 365.78515625# disk.nvme0c0n1.in‚ 10916.5546875) disk./.usagePercent‚39.45864861855877 disk.dm-0.in‚ 10916.5546875 disk.drbd2165.out‚0 disk.sda.in‚ 0.2421875 disk.sda.out‚ 365.78515625* proc.memory.percent‚17.505192756652832 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚88.30254077911377*Ø ”•‚(:‰  ËêÒ¡Ðÿ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pdÌ.–‚(:£  ËêÒ™Ñþ®$ disk./.usageGB‚587.9624404907227 disk.sda3.in‚ 0.2421875 disk.sda2.in‚0# disk.rbd1.out‚17768.11474609375 disk.dm-1.in‚ 0.2421875# proc.memory.rssMB‚ 2868.05078125) disk./.usagePercent‚39.45865271467932 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd2.out‚ 37.517578125* proc.memory.percent‚17.505192756652832$ disk.nvme0c0n1.in‚10917.38671875" disk.nvme0n1.in‚10917.38671875 disk.sda.out‚ 366.05859375 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 61634.828125! disk.nvme0n1.out‚ 61634.828125 disk.rbd0.out‚0 disk.drbd2165.in‚0 network.sent‚ 6.472105e+06 disk.rbd1.in‚ 506.4921875 disk.drbd2165.out‚0 disk.dm-1.out‚ 366.05859375 cpu‚25.513664056279286 network.recv‚ 1.480463e+06 disk.sda3.out‚ 366.05859375 disk.sda.in‚ 0.2421875 disk.rbd2.in‚0.015625 disk.dm-0.in‚10917.38671875 disk.dm-0.out‚ 61634.828125 disk.sda1.out‚0$ memory_percent‚88.30258846282959) proc.memory.availableMB‚ 1916.50390625 proc.cpu.threads‚17‘”¸™2—- _wandbruntime‚5801  _runtime‚5801ý²‡ßƘÀ $ _timestamp‚1.784325548585832e+09 _runtime‚5801.960432438 _step‚284 & lr-AdamW/pg1‚0.0002575057167215396 & lr-AdamW/pg2‚0.0002575057167215396 trainer/global_step‚3499J´Èø˙Š& lr-AdamW/pg1‚0.0002575057167215396 & lr-AdamW/pg2‚0.0002575057167215396 trainer/global_step‚3499 $ _timestamp‚1.784325548585832e+09 _runtime‚5801.960432438 _step‚284œ`ôç2š- _wandbruntime‚5801  _runtime‚5801&ôiÆû'›õO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3283870348895 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30280774 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843255485869436e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3289496559616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052214 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007002034109440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1003713882939392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _step‚285 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1003713909112832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚3499 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1003713909112832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29781419 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1003280008908800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30270801 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8976043 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29781413 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3282525748573 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29781413 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9933657 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30270801 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1006562509229405 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007002034109440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18909464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276022048516608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003279983480832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271779076214784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29781419 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8976048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4242604210688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30280774 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007003405672448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3288151170048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276021647986176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3288151170048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062193 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271779444305920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007003405672448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1003713882939392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3289496559616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9933421 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1006563879257695 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052214 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007002034109440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1003713882939392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _step‚285 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_star.…™#!t/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1003713909112832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚3499 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1003713909112832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29781419 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1003280008908800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30270801 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8976043 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29781413 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3282525748573 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29781413 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9933657 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30270801 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1006562509229405 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007002034109440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18909464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276022048516608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003279983480832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271779076214784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29781419 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8976048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4242604210688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30280774 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007003405672448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3288151170048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276021647986176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3288151170048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062193 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271779444305920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007003405672448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1003713882939392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3289496559616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9933421 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1006563879257695 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1003991909842944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18914753 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚5803.350936881 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3289076205568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4243765495296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3283449214819 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29789625 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30279279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276100728641024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1003991883669504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8978591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 $ _timestamp‚1.784325549976155e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007282331445248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8978586 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9936389 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4243797927424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30289266 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30279279 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078891 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30289266 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3290421602304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1006841314473827 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29789619 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276100328117760 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1003557890686976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068898 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078891 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9936167 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚286 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1006842684502229 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068898 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078891 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9936167 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚286 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1006842684502229 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1003991909842944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18914753 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚5803.350936881 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3289076205568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4243765495296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3283449214819 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29789625 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30279279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276100728641024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1003991883669504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8978591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 $ _timestamp‚1.784325549976155e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007282331445248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8978586 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9936389 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4243797927424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30289266 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30279279 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078891 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30289266 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440ž £Õ2 - _wandbruntime‚5803  _runtime‚5803¹ýØ¡ /loss/FRt1/detection_loss‚1.5053070458468427 0loss/SRqqt2/detection_loss‚0.936194255630501 % _timestamp‚1.7843255500607843e+09 _runtime‚5803.435122019 0loss/FRt2/assignment_loss‚1.8402296216485563 2loss/FBt2/assignment_loss‚0.004352097743696899 /loss/FRt2/detection_loss‚0.5984278959946333 /loss/FBt2/detection_loss‚1.1811185993719846 _step‚287 2loss/SRqqt1/assignment_loss‚0.5382218248987911 2loss/FBt1/assignment_loss‚0.040844723993869056 %loss/total_loss‚9.205686174481794 epoch‚8 trainer/global_step‚3499 1loss/SRqqt1/detection_loss‚0.6366189876615169 /loss/FBt1/detection_loss‚0.5268538441265178 /loss/FRt1/assignment_loss‚0.759822897964649 2loss/SRqqt2/assignment_loss‚0.6376943796002363b:F– ¢† 2loss/SRqqt1/assignment_loss‚0.5382218248987911 2loss/FBt1/assignment_loss‚0.040844723993869056 %loss/total_loss‚9.205686174481794 epoch‚8 trainer/global_step‚3499 1loss/SRqqt1/detection_loss‚0.6366189876615169 /loss/FBt1/detection_loss‚0.5268538441265178 /loss/FRt1/assignment_loss‚0.759822897964649 2loss/SRqqt2/assignment_loss‚0.6376943796002363 /loss/FRt1/detection_loss‚1.5053070458468427 0loss/SRqqt2/detection_loss‚0.936194255630501 % _timestamp‚1.7843255500607843e+09 _runtime‚5803.435122019 0loss/FRt2/assignment_loss‚1.8402296216485563 2loss/FBt2/assignment_loss‚0.004352097743696899 /loss/FRt2/detection_loss‚0.5984278959946333 /loss/FBt2/detection_loss‚1.1811185993719846 _step‚287Ÿ¦¬ØøŸ£‚(:” ¯ËêҌ汭 gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚300.279' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.055800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;‘#n)¤‚(:ž ¯ËêÒúíµ®' disk.nvme0c0n1.out‚61676.9619140625 disk.rbd2.out‚ 37.517578125 proc.cpu.threads‚17 disk.sda1.in‚0) disk./.usagePercent‚39.46676943355241 disk.drbd2165.out‚0 disk.sda3.in‚ 0.2421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.sda.in‚ 0.2421875 disk.sda.out‚ 368.6171875 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2868.05078125 cpu‚25.514148657053003% disk.nvme0n1.out‚61676.9619140625 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 10926.296875# memory_percent‚88.2965087890625# disk.rbd1.out‚17768.11474609375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 368.6171875" proc.memory.availableMB‚1917.5* proc.memory.percent‚17.505192756652832 disk.sda3.out‚ 368.6171875 network.sent‚ 6.477267e+06$ disk./.usageGB‚588.0833854675293 disk.dm-0.in‚ 10926.296875" disk.dm-0.out‚61676.9619140625 network.recv‚ 1.484281e+06" disk.nvme0c0n1.in‚ 10926.296875a"ôž¥‚(:“ ¾ËêÒâÇŠ¬ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.199' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.03980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãH®¦¦‚(:¦ ¾ËêÒâÔ·¯ disk.sda2.out‚0* disk./.usagePercent‚39.475679007759034 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 370.3984375# proc.memory.rssMB‚ 2868.05078125! disk.dm-0.out‚61714.478515625 disk.dm-1.in‚ 0.2421875 disk.sda3.in‚ 0.2421875 disk.drbd2165.out‚0 disk.rbd1.in‚ 506.4921875 network.recv‚ 1.489354e+06$ disk.nvme0n1.out‚61714.478515625& disk.nvme0c0n1.out‚61714.478515625 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 10930.3828125) proc.memory.availableMB‚ 1917.32421875# disk.rbd1.out‚17768.17724609375 network.sent‚ 6.540436e+06 disk.sda.in‚ 0.2421875# di›ŠÞ‹sk.nvme0c0n1.in‚ 10930.3828125! disk.nvme0n1.in‚ 10930.3828125$ memory_percent‚88.29758167266846* proc.memory.percent‚17.505192756652832 disk.drbd2165.in‚0$ disk./.usageGB‚588.2161445617676 disk.sda.out‚ 370.3984375 disk.sda2.in‚0 disk.rbd2.out‚ 37.517578125 disk.sda1.in‚0 disk.dm-1.out‚ 370.3984375 cpu‚25.51483026214471 proc.cpu.threads‚17ÌdŸ§‚(:” ÍËêÒÞÆÛ¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rbŠ4¨‚(:© ÍËêÒ™â߯ network.recv‚ 1.49294e+06) disk./.usagePercent‚39.47248352171665# disk./.usageGB‚588.168529510498# disk.dm-0.out‚61788.52099609375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 11005.6484375( disk.nvme0c0n1.out‚61788.52099609375 disk.sda2.in‚0 disk.dm-0.in‚ 11005.6484375 disk.sda1.out‚0 disk.dm-1.out‚ 371.1171875# disk.rbd1.out‚17768.17724609375 cpu‚25.515344110821637 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 371.1171875 disk.rbd1.in‚ 506.4921875$ memory_percent‚88.29512596130371* proc.memory.percent‚17.505192756652832 disk.sda3.in‚ 0.2421875( proc.memory.availableMB‚ 1917.7265625# proc.memory.rssMB‚ 2868.05078125 disk.rbd2.in‚0.015625 disk.sda.in‚ 0.2421875 disk.sda.out‚ 371.1171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.2421875! disk.nvme0n1.in‚ 11005.6484375 network.sent‚ 6.546137e+06 disk.rbd2.out‚ 37.517578125 disk.sda2.out‚0& disk.nvme0n1.out‚61788.52099609375 proc.cpu.threads‚17Â}6’©‚(:‡ ÜËêҾ࿭ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚342.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.586 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oy”8ª‚(:­ ÜËêҚ‹° disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 11006.0859375# disk.dm-0.out‚61857.70166015625 disk.dm-1.in‚ 0.2421875# disk.nvme0c0n1.in‚ 11006.0859375 disk.sda2.in‚0) disk./.usagePercent‚39.46925936283043 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚61857.70166015625# disk.rbd1.out‚17768.23974609375 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.in‚ 506.4921875 cpu‚25.516025380499887 disk.sda1.out‚0! disk.nvme0n1.in‚ 11006.0859375# memory_percent‚88.2967472076416* proc.memory.percent‚17.505192756652832 network.sent‚ 6.550704e+06 network.recv‚ 1.496459e+06$ disk./.usageGB‚588.1204872131348 disk.rbd2.out‚ 37.521484375# proc.memory.rssMB‚ 2868.05078125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 371.81640625 disk.sda3.out‚ 371.81640625 disk.sda.in‚ 0.2421875 disk.sda.out‚ 371.81640625( proc.memory.availableMB‚ 1917.4609375 disk.sda3.in‚ 0.2421875& disk.nvme0n1.out‚61857.70166015625£Ð¥”«‚(:‰ ëËêÒíݬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.9504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i€1¬‚(:¦ ëËêÒÛ׉¯ disk.sda3.in‚ 0.2421875$ disk.nvme0c0n1.in‚11007.04296875& disk.nvme0c0n1.out‚61872.173828125 disk.dm-1.in‚ 0.2421875 disk.rbd2.in‚0.015625 disk.rbd1.in‚ 506.4921875# proc.memory.rssMB‚ 2868.05078125 disk.dm-0.in‚11007.04296875 disk.sda.in‚ 0.2421875 disk.sda.out‚ 372.0859375& proc.memory.availableMB‚ 1917.21875 cpu‚25.516587822310818 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.sent‚ 6.55577e+06 network.recv‚ 1.500032e+06 disk.sda1.in‚0 disk.sda3.out‚ 372.0859375# disk.rbd1.out‚17768.23974609375* proc.memory.percent‚17.505192756652832 disk.drbd2165.in‚0 disk.dm-1.out‚ 372.0859375" disk.nvme0n1.in‚11007.04296875 disk.sda2.in‚0 disk.sda2.out‚0* disk./.usagePercent‚39.469261922905766 disk.sda1.out‚0$ disk.nvme0n1.out‚61872.173828125$ memory_percent‚88.29822540283203$ disk./.usageGB‚588.1205253601074 disk.rbd2.out‚ 37.521484375! disk.dm-0.out‚61872.173828125F ¨ý”­‚(:‰ úËêÒúŠÚ¬ gpu.0.gpu‚51.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚321.162' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.2324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’B Û0®‚(:¥ úËêÒàçà® disk.sda.out‚ 372.359375 disk.sda3.in‚ 0.2421875" disk.nvme0n1.in‚11007.04296875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚11007.04296875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0$ memory_percent‚88.29677104949951 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 network.sent‚ 6.560869e+06 disk.dm-1.out‚ 372.359375 disk.sda2.in‚0 cpu‚25.51722593976233 network.recv‚ 1.503499e+06$ disk./.usageGB‚588.1209144592285# proc.memory.rssMB‚ 2868.05078125* proc.memory.percent‚17.505192756652832 disk.sda1.in‚0) proc.memory.availableMB‚ 1917.45703125# disk.rbd1.out‚17768.30224609375! disk.dm-0.out‚61877.845703125 disk.sda.in‚ 0.2421875) disk./.usagePercent‚39.46928803567426$ disk.nvme0c0n1.in‚11007.04296875& disk.nvme0c0n1.out‚61877.845703125$ disk.nvme0n1.out‚61877.845703125 disk.dm-1.in‚ 0.2421875 disk.sda3.out‚ 372.359375 disk.rbd2.out‚ 37.5839843757ŧ2¯- _wandbruntime‚5881  _runtime‚5881@oû~Æ°À _step‚288 & lr-AdamW/pg1‚0.0002569784983200006 & lr-AdamW/pg2‚0.0002569784983200006 trainer/global_step‚3549 $ _timestamp‚1.784325627935817e+09 _runtime‚5881.310479968÷é}f˱Š_runtime‚5881.310479968 _step‚288 & lr-AdamW/pg1‚0.0002569784983200006 & lr-AdamW/pg2‚0.0002569784983200006 trainer/global_step‚3549 $ _timestamp‚1.784325627935817e+09 È06A2²- _wandbruntime‚5881  _runtime‚5881Æò(ý'³÷O YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3330107462003 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843256279369805e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10071698 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020948386545664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017613945618432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5881.311727122 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30191713 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279956579397120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017613919444992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19174652 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888266 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚289 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9103198 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9103193 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19174896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020949758108672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19174896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020949758108672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10071698 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020948386545664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017613945618432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5881.311727122 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30191713 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279956579397120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017613919444992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19174652 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888266 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973¡Cíüm2µ- _wandbruntime‚5882  _runtime‚5882¬혲¶N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3335392136192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9105741 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020781640495735 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017891920175104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚3549 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280034869418496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275731251097600 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30199925 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017891946348544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336737532928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017891920175104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020783010524137 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275730883006464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30705031 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4303986412032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 % _timestamp‚1.7843256293221116e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚290 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19179900 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_poolÛ’ó3U.freed‚10074164 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30715018 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30199919 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1021228683881472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336737532928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10074386 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4304018844160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1017451979595776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9105736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30715018 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30199919 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1021228683881472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336737532928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10074386 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4304018844160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1017451979595776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9105736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30199925 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017891946348544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336737532928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017891920175104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020783010524137 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275730883006464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30705031 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4303986412032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 % _timestamp‚1.7843256293221116e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚290 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19179900 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10074164 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926¢Á°2¸- _wandbruntime‚5882  _runtime‚5882ÿùŸä ¹ƒ 0loss/FRt2/assignment_loss‚2.0009938512659895 2loss/SRqqt1/assignment_loss‚0.5497398547226332 2loss/SRqqt2/assignment_loss‚0.5464241650973292 /loss/FRt1/detection_loss‚1.5286992797097125 /loss/FBt1/detection_loss‚0.4922366332388125 /loss/FBt2/detection_loss‚0.5899548039843256 _step‚291 2loss/FBt1/assignment_loss‚0.026126574050315116 .loss/FRt2/detection_loss‚0.608713970033393 1loss/SRqqt1/detection_loss‚0.7211282481178359 1loss/SRqqt2/detection_loss‚1.0135320356139967 epoch‚8 % _timestamp‚1.7843256294086041e+09 0loss/FRt1/assignment_loss‚0.7841207100489926 3loss/FBt2/assignment_loss‚0.0060446387491048465 %loss/total_loss‚8.867714764632442 trainer/global_step‚3549 _runtime‚5882.782925678EK] ºˆ /loss/FRt1/detection_loss‚1.5286992797097125 /loss/FBt1/detection_loss‚0.4922366332388125 /loss/FBt2/detection_loss‚0.5899548039843256 _step‚291 2loss/FBt1/assignment_loss‚0.026126574050315116 .loss/FRt2/detection_loss‚0.608713970033393 1loss/SRqqt1/detection_loss‚0.7211282481178359 1loss/SRqqt2/detection_loss‚1.0135320356139967 epoch‚8 % _timestamp‚1.7843256294086041e+09 0loss/FRt1/assignment_loss‚0.7841207100489926 3loss/FBt2/assignment_loss‚0.0060446387491048465 %loss/total_loss‚8.867714764632442 trainer/global_step‚3549 _runtime‚5882.782925678 0loss/FRt2/assignment_loss‚2.0009938512659895 2loss/SRqqt1/assignment_loss‚0.5497398547226332 2loss/SRqqt2/assignment_loss‚0.5464241650973292£¼¿?”»‚(:‰ ‰ÌêÒ…êЮ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬÷48¼‚(:­ ‰ÌêÒÍÉÖ° proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚88.29813003540039 cpu‚25.517714001548104 disk.rbd0.out‚0# disk.dm-0.out‚61889.57470703125 disk.sda3.out‚ 372.74609375' proc.memory.availableMB‚ 1917.234375* proc.memory.percent‚17.505192756652832 disk.sda.out‚ 372.74609375# proc.memory.rssMB‚ 2868.05078125 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.dm-1.out‚ 372.74609375( disk.nvme0c0n1.out‚61889.57470703125& disk.nvme0n1.out‚61889.57470703125 disk.sda2.in‚0 network.recv‚ 1.507063e+06* disk./.usagePercent‚39.469206625278375# disk./.usageGB‚588.119701385498 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.2421875# disk.nvme0c0n1.in‚ 11277.7890625 disk.sda.in‚ 0.2421875! disk.nvme0n1.in‚ 11277.7890625 disk.sda2.out‚0 network.sent‚ 6.565926e+06 disk.rbd1.in‚ 506.4921875# disk.rbd1.out‚17768.30224609375 disk.rbd2.out‚ 37.583984375 disk.dm-1.in‚ 0.2421875 disk.dm-0.in‚ 11277.7890625I„Ð딽‚(:‰ ˜ÌêÒØ©ç« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‰K“5¾‚(:ª ˜ÌêÒ°û­" disk.nvme0n1.in‚11278.74609375 network.sent‚ 6.629109e+06$ disk./.usageGB‚588.1198310852051 disk.drbd2165.out‚0 disk.dm-1.out‚ 373.328125 cpu‚25.51807277879266 disk.rbd2.in‚0.015625& disk.nvme0n1.out‚61899.06494140625 disk.sda.out‚ 373.328125$ memory_percent‚88.29693794250488 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1917.4296875# proc.memory.rssMB‚ 2868.05078125 disk.sda2.out‚0# disk.rbd1.out‚17768.36474609375 disk.sda1.in‚0 disk.dm-1.in‚ 0.2421875$ disk.nvme0c0n1.in‚11278.74609375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.2421875 disk.sda2.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚39.469215329534535 disk.dm-0.in‚11278.74609375# disk.dm-0.out‚61899.06494140625 disk.sda.in‚ 0.2421875 network.recv‚ 1.512053e+06 disk.rbd1.in‚ 506.4921875 disk.rbd2.out‚ 38.37109375 disk.sda3.out‚ 373.328125* proc.memory.percent‚17.505192756652832 proc.cpu.threads‚17( disk.nvme0c0n1.out‚61899.06494140625 óL}”¿‚(:‰ §ÌêÒ•ÂÙ« gpu.0.gpu‚77.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…÷;+À‚(:  §ÌêÒ²žä­ disk.sda3.out‚374.0625 disk.rbd0.out‚0 disk.dm-1.out‚374.0625 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 38.37109375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.2421875) disk./.usagePercent‚39.46948004132494$ disk./.usageGB‚588.1237754821777 disk.dm-0.in‚11278.74609375 disk.sda.out‚374.0625 disk.rbd0.in‚0$ memory_percent‚88.29727172851562$ proc.memory.availableMB‚1917.375# proc.memory.rssMB‚ 2868.05078125 disk.sda2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17# disk.rbd1.out‚17768.36474609375 network.recv‚ 1.515568e+06 disk.drbd2165.out‚0 cpu‚25.518785142410138# disk.dm-0.out‚61906.28466796875 disk.sda3.in‚ 0.2421875$ disk.nvme0c0n1.in‚11278.74609375& disk.nvme0n1.out‚61906.28466796875 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875* proc.memory.percent‚17.505192756652832 network.sent‚ 6.634704e+06( disk.nvme0c0n1.out‚61906.28466796875" disk.nvme0n1.in‚11278.74609375 disk.sda.in‚ 0.2421875EÓM”Á‚(:‰ ¶ÌêÒÔ¶¬ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øˆ3‚(:¨ ¶ÌêÒÕÒ¤¯ disk.dm-0.in‚ 11279.578125* proc.memory.percent‚17.505192756652832 disk.rbd2.in‚0.015625! disk.dm-0.out‚61917.912109375 disk.drbd2165.out‚0 disk.dm-1.out‚ 374.60546875$ disk./.usageGB‚588.1240577697754 disk.drbd2165.in‚0 cpu‚25.51926934060043 proc.cpu.threads‚17 disk.dm-1.in‚ 0.2421875 disk.sda3.in‚ 0.2421875# proc.memory.rssMB‚ 2868.05078125$ disk.nvme0n1.out‚61917.912109375 disk.sda.out‚ 374.60546875* disk./.usagePercent‚39.469498985882474 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 11279.578125$ memory_percent‚88.29729557037354) proc.memory.availableMB‚ 1917.37109375& disk.nvme0c0n1.out‚61917.912109375 network.recv‚ 1.518965e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚38.41455078125 network.sent‚ 6.639903e+06 disk.sda.in‚ 0.2421875 disk.sda1.in‚0 disk.sda3.out‚ 374.60546875" disk.nvme0c0n1.in‚ 11279.578125 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875# disk.rbd1.out‚17768.42724609375*Õ*ҞÂ(:“ ÅÌêÒ¥ú’­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚299.167' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.83339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—<¹`7Ä‚(:¬ ÅÌêÒÎÍ…° disk.rbd2.out‚38.41845703125 disk.dm-1.out‚ 374.828125$ disk.nvme0c0n1.in‚11313.87890625 disk.sda.in‚ 0.2421875# proc.memory.rssMB‚ 2868.05078125 cpu‚25.519757872506787 disk.rbd0.in‚0 network.sent‚ 6.644418e+06 disk.sda1.in‚0& disk.nvme0n1.out‚61929.40771484375 disk.sda.out‚ 374.828125# disk.dm-0.out‚61929.40771484375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 506.4921875# memory_percent‚88.2988691329956 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.2421875 disk.sda3.in‚ 0.2421875 disk.sda2.out‚0 disk.sda3.out‚ 374.828125# disk.rbd1.out‚17768.42724609375* proc.memory.percent‚17.505192756652832 network.recv‚ 1.522505e+06) disk./.usagePercent‚39.46949181767152 disk.dm-0.in‚11313.87890625 disk.drbd2165.out‚0" disk.nvme0n1.in‚11313.87890625) proc.memory.availableMB‚ 1917.11328125 proc.cpu.threads‚17# disk./.usageGB‚588.123950958252( disk.nvme0c0n1.out‚61929.40771484375 disk.sda2.in‚0 disk.rbd0.out‚0 Ç'}2Å- _wandbruntime‚5961  _runtime‚5961#° _ÈÆ $ _timestamp‚1.784325708467964e+09 _runtime‚5961.842605719 _step‚292 ' lr-AdamW/pg1‚0.00025644357605639624 ' lr-AdamW/pg2‚0.00025644357605639624 trainer/global_step‚3599ã&ÓÍÇÇ trainer/global_step‚3599 $ _timestamp‚1.784325708467964e+09 _runtime‚5961.842605719 _step‚292 ' lr-AdamW/pg1‚0.00025644357605639624 ' lr-AdamW/pg2‚0.00025644357605639624¤‰3k2È- _wandbruntime‚5961  _runtime‚5961fÌÚá ÉöO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3382128420864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1031068186726400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3374999974789 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376344575111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1034896110544896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19439502 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚293 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724318 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31122305 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279527716982784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19439261 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283891147809280 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1031068161298432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30602019 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9230343 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283890747278848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1034894738981888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 cQDeviceStatsMonitor.on_train_baS-í¸tch_start/inactive_split_bytes.small_pool.allocated‚ 4363062735360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5961.843905743 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10209154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31122305 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031513955950592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3382128420864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034443161273221 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175S·8þ(ÊûO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3382128420864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1031068186726400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3374999974789 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376344575111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1034896110544896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19439502 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚293 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724318 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31122305 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279527716982784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19439261 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283891147809280 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1031068161298432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30602019 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9230343 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283890747278848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1034894738981888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4363062735360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5961.843905743 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10209154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31122305 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031513955950592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3382128420864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034443161273221 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648¥]S2m2Ë- _wandbruntime‚5963  _runtime‚5963!Ä€a'ÌN FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1035173664747520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1031346043076608 _step‚294 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31130783 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1031791982854144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1035175036317696 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1031791956680704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4364260457984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61750995 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61750995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31140770 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30610225 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843257098668272e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283969831939584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19444564 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377268041469 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9232886 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3381708066816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605571481600 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3375923441035 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10211678 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5963.241639834 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3383053463552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30610219 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283969431416320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1031791982854144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0júÕ} '͆N QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3383053463552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30610219 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283969431416320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1031791982854144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1035173664747520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1031346043076608 _step‚294 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31130783 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1031791982854144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1035175036317696 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1031791956680704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4364260457984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61750995 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61750995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31140770 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30610225 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843257098668272e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283969831939584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19444564 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377268041469 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9232886 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3381708066816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605571481600 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3375923441035 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10211678 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5963.241639834¦ÁÊ™²2Î- _wandbruntime‚5963  _runtime‚5963wMfü<Ï€ 0loss/FRt1/assignment_loss‚0.7874121629830563 0loss/FBt1/detection_loss‚0.41317414387996343 /loss/FBt2/detection_loss‚0.5543720167130232 % _timestamp‚1.7843257099500418e+09 _runtime‚5963.324425918 1loss/FBt2/assignment_loss‚0.17743570443567383 /loss/FRt1/detection_loss‚1.5060272652282443 trainer/global_step‚3599 /loss/FRt2/detection_loss‚0.5821796649101363 1loss/SRqqt2/detection_loss‚1.0081761709209216 epoch‚8 _step‚295 /loss/FRt2/assignment_loss‚1.818656730984586 2loss/SRqqt1/assignment_loss‚0.544085;Ñ» Ê3312071938 1loss/SRqqt2/assignment_loss‚0.558671561865357 1loss/FBt1/assignment_loss‚0.06582993672204798 1loss/SRqqt1/detection_loss‚0.6797929106727294 %loss/total_loss‚8.695813600522934©BL Ð… /loss/FRt2/assignment_loss‚1.818656730984586 2loss/SRqqt1/assignment_loss‚0.5440853312071938 1loss/SRqqt2/assignment_loss‚0.558671561865357 1loss/FBt1/assignment_loss‚0.06582993672204798 1loss/SRqqt1/detection_loss‚0.6797929106727294 %loss/total_loss‚8.695813600522934 0loss/FRt1/assignment_loss‚0.7874121629830563 0loss/FBt1/detection_loss‚0.41317414387996343 /loss/FBt2/detection_loss‚0.5543720167130232 % _timestamp‚1.7843257099500418e+09 _runtime‚5963.324425918 1loss/FBt2/assignment_loss‚0.17743570443567383 /loss/FRt1/detection_loss‚1.5060272652282443 trainer/global_step‚3599 /loss/FRt2/detection_loss‚0.5821796649101363 1loss/SRqqt2/detection_loss‚1.0081761709209216 epoch‚8 _step‚295§`8ؔт(:‰ ÔÌêÒ–âѬ gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Š¯6Ò‚(:“ ÔÌêÒÿ â¯# disk./.usageGB‚588.125659942627 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 375.15625) proc.memory.availableMB‚ 1915.88671875 disk.rbd1.in‚ 506.4921875# disk.rbd1.out‚17768.55224609375 disk.dm-0.out‚ 61933.5625 disk.sda3.in‚ 0.2421875! disk.nvme0c0n1.out‚ 61933.5625* proc.memory.percent‚17.513251304626465 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚11496.39453125 disk.sda.in‚ 0.2421875 disk.sda.out‚ 375.15625 cpu‚25.52288020736403 proc.cpu.threads‚17 disk.rbd2.out‚38.72314453125$ disk.nvme0c0n1.in‚11496.39453125# memory_percent‚88.3063554763794# proc.memory.rssMB‚ 2869.37109375 network.recv‚ 1.525849e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 375.15625 disk.sda2.in‚0 network.sent‚ 6.64962e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.out‚ 61933.5625* disk./.usagePercent‚39.469606509046855 disk.rbd0.out‚0" disk.nvme0n1.in‚11496.39453125 disk.rbd0.in‚0 disk.sda1.in‚0Y%o“Ó‚(:ˆ ãÌêÒ²Œ„¬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚313.635' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚62.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô«R6Ô‚(:« ãÌêÒŠ£å® disk.rbd2.out‚38.72314453125 cpu‚25.52936262831224 disk.dm-1.out‚ 376.9921875 network.sent‚ 6.712974e+06# disk.dm-0.out‚61955.11181640625 disk.sda.out‚ 376.9921875# disk.rbd1.out‚17768.55224609375# memory_percent‚88.3948564529419) proc.memory.availableMB‚ 1901.38671875 proc.cpu.threads‚17 network.recv‚ 1.531078e+06 disk.rbd0.in‚0$ disk./.usageGB‚588.1299362182617 disk.dm-0.in‚ 11497.2265625# disk.nvme0c0n1.in‚ 11497.2265625! disk.nvme0n1.in‚ 11497.2265625 disk.rbd1.in‚ 506.4921875) disk./.usagePercent‚39.46989349349274 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.sda3.out‚ 376.9921875" proc.memory.rssMB‚ 2883.8828125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.2421875* proc.memory.percent‚17.601823806762695( disk.nvme0c0n1.out‚61955.11181640625& disk.nvme0n1.out‚61955.11181640625 disk.sda.in‚ 0.2421875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.2421875 disk.sda2.in‚0Gƒu<2Õ- _wandbruntime‚5990  _runtime‚5990ø÷Aß=ÖÙ{ ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43759772181510925 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813916087150574 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.25601357221603394 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1622692346572876 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970422387123108 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967845678329468 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13542498648166656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909666776657104 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39263805747032166 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14916756749153137 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6104199290275574 trainer/global_step‚3599 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6073773503303528 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7798563838005066 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871394038200378 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840995073318481 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09448689222335815 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5246391892433167 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1506451964378357 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995954692363739 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983595609664917 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7872260808944702 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15602633357048035 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5338982939720154 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10232418775558472 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019443247467279434 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09815578162670135 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11444021761417389 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968254566192627 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6373830437660217 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2307746559381485 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5284346342086792 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 epoch‚8 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5647713541984558 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06651818007230759 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.919143795967102 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5218959450721741 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26096054911613464 +jet/accuracy_3_of_4‚0.08398675918579102 /particle/accuracy_4_of_4‚0.9434896111488342 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.029782351106405258 particle/accuracy_2_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14014345407485962 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4860932528972626 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.169310063123703 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11062003672122955 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.657660722732544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4605148732662201 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5813398361206055 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13542498648166656 _step‚296 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7268166542053223 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1300622969865799 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6354434490203857 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1928153932094574 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15261200070381165 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5390822291374207 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46772122383117676 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7805744409561157 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5758333206176758 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9041048884391785 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8964375257492065 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8449767231941223 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4937807321548462 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1554110199213028 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010787810198962688 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20833174884319305 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9881786108016968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1029672846198082 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.48528239130973816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.46190571784973145 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_1_of_1‚0.24610449373722076 /particle/accuracy_3_of_3‚0.9995090961456299 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920070171356201 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.535041868686676 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8585860729217529 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13047370314598083 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6692708134651184 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 % _timestamp‚1.7843257375211105e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953017234802246 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915674924850464 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1728179156780243 ,jet/accuracy_4_of_4‚0.013899499550461769 'particle/f_score‚0.6198922395706177 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6981551647186279 *jet/accuracy_1_of_4‚0.9429936408996582 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9681989550590515 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4931234121322632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6035695672035217 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6302308440208435 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1029672846198082 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7260165214538574 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6179987788200378 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28362587094306946 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7181028127670288 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15395523607730865 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934422373771667 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6438726186752319 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24861036241054535 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7805744409561157 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940187335014343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930242896080017 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967845678329468 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15004055202007294 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7533458471298218 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4977996349334717 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4860932528972626 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01556957047432661 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 )loss/val_total_loss‚8.868798688767741 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5826659798622131 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7812547087669373 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005960495211184025 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_4‚0.6876555681228638 7validation_average_jet_accuracy‚0.29455190896987915 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2883451581001282 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18346135318279266 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11273409426212311 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 _runtime‚ 5990.89640071 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09456966817378998 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13513648509979248 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8402391076087952 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.656839907169342 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5755531787872314 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5660060048103333 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09808117896318436 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5784317255020142 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5688104629516602 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5046296119689941 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6712336540222168 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5779283046722412 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5826769471168518 jet/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14148060977458954 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9926702380180359 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21082976460456848 *jet/accuracy_1_of_2‚0.6919612884521484 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2351021021604538 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17533142864704132 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_4‚0.9904478192329407 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_2_of_2‚0.11924450844526291 )jet/accuracy_1_of_3‚0.882352352142334 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4927273690700531 particle/accuracy_1_of_2‚1 +particle/sensitivity‚0.6066792011260986 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9117778539657593 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5280225276947021 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6981551647186279 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967204332351685 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15666666626930237 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029039127752184868 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.43739086389541626 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5900510549545288f9`2×Þ{ ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6981551647186279 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967204332351685 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15666666626930237 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029039127752184868 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.43739086389541626 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5900510549545288 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43759772181510925 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813916087150574 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.25601357221603394 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1622692346572876 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970422387123108 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967845678329468 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13542498648166656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909666776657104 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39263805747032166 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14916756749153137 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6104199290275574 trainer/global_step‚3599 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6073773503303528 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7798563838005066 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871394038200378 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840995073318481 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09448689222335815 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5246391892433167 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1506451964378357 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995954692363739 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983595609664917 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7872260808944702 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15602633357048035 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5338982939720154 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10232418775558472 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019443247467279434 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09815578162670135 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11444021761417389 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968254566192627 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6373830437660217 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2307746559381485 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5284346342086792 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 epoch‚8 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5647713541984558 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06651818007230759 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.919143795967102 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5218959450721741 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26096054911613464 +jet/accuracy_3_of_4‚0.08398675918579102 /particle/accuracy_4_of_4‚0.9434896111488342 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.029782351106405258 particle/accuracy_2_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14014345407485962 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4860932528972626 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.169310063123703 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11062003672122955 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.657660722732544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4605148732662201 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5813398361206055 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13542498648166656 _step‚296 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7268166542053223 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1300622969865799 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6354434490203857 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1928153932094574 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15261200070381165 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5390822291374207 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46772122383117676 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7805744409561157 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5758333206176758 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9041048884391785 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8964375257492065 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8449767231941223 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4937807321548462 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1554110199213028 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010787810198962688 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20833174884319305 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9881786108016968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1029672846198082 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.48528239130973816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.46190571784973145 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_1_of_1‚0.24610449373722076 /particle/accuracy_3_of_3‚0.9995090961456299 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920070171356201 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.535041868686676 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8585860729217529 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13047370314598083 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6692708134651184 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 % _timestamp‚1.7843257375211105e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953017234802246 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915674924850464 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1728179156780243 ,jet/accuracy_4_of_4‚0.013899499550461769 'particle/f_score‚0.6198922395706177 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6981551647186279 *jet/accuracy_1_of_4‚0.9429936408996582 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9681989550590515 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4931234121322632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6035695672035217 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6302308440208435 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1029672846198082 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7260165214538574 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6179987788200378 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28362587094306946 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7181028127670288 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15395523607730865 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934422373771667 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6438726186752319 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24861036241054535 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7805744409561157 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940187335014343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930242896080017 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967845678329468 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15004055202007294 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7533458471298218 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4977996349334717 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4860932528972626 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01556957047432661 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 )loss/val_total_loss‚8.868798688767741 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5826659798622131 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7812547087669373 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005960495211184025 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_4‚0.6876555681228638 7validation_average_jet_accuracy‚0.29455190896987915 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2883451581001282 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18346135318279266 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11273409426212311 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 _runtime‚ 5990.89640071 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09456966817378998 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.135136485099792Р𣄠48 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8402391076087952 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.656839907169342 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5755531787872314 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5660060048103333 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09808117896318436 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5784317255020142 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5688104629516602 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5046296119689941 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6712336540222168 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5779283046722412 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5826769471168518 jet/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14148060977458954 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9926702380180359 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21082976460456848 *jet/accuracy_1_of_2‚0.6919612884521484 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2351021021604538 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17533142864704132 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_4‚0.9904478192329407 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_2_of_2‚0.11924450844526291 )jet/accuracy_1_of_3‚0.882352352142334 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4927273690700531 particle/accuracy_1_of_2‚1 +particle/sensitivity‚0.6066792011260986 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9117778539657593 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5280225276947021 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 particle/accuracy_5_of_5‚0¨®0z”Ø‚(:‰ òÌêÒŒ»ß« gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%¼9Ù‚(:® òÌêÒà˜ã®# disk.rbd1.out‚17768.61474609375 disk.rbd2.out‚ 38.794921875" disk.dm-0.out‚61962.7177734375 disk.sda.in‚ 0.2421875) proc.memory.availableMB‚ 1894.39453125 cpu‚25.532484800102942 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 377.39453125' disk.nvme0c0n1.out‚61962.7177734375$ memory_percent‚88.43753337860107# proc.memory.rssMB‚ 2890.48046875 disk.rbd2.in‚0.015625 disk.dm-0.in‚11498.18359375 disk.drbd2165.in‚0) proc.memory.percent‚17.64209270477295 proc.cpu.threads‚17 network.sent‚ 6.718583e+06) disk./.usagePercent‚39.46992523842698% disk.nvme0n1.out‚61962.7177734375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚11498.18359375" disk.nvme0n1.in‚11498.18359375 disk.sda.out‚ 377.39453125 disk.rbd1.in‚ 506.4921875 disk.sda3.in‚ 0.2421875 network.recv‚ 1.534409e+06 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.2421875 disk.sda3.out‚ 377.39453125$ disk./.usageGB‚588.1304092407227 Ä™~”Ú‚(:‰ ÍêÒ앤¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.8552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uGŸø3Û‚(:¨ ÍêÒÝùÍ® network.recv‚ 1.539588e+06 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚11498.18359375 disk.sda.out‚ 379.6640625 disk.sda2.in‚0& disk.nvme0c0n1.out‚61975.923828125$ disk.nvme0n1.out‚61975.923828125 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.rbd2.out‚ 38.794921875$ memory_percent‚88.43750953674316 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 379.6640625 disk.sda3.in‚ 0.2421875 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2890.48046875$ disk./.usageGB‚588.1273956298828# disk.rbd1.out‚17768.61474609375) proc.memory.percent‚17.64209270477295 disk.sda.in‚ 0.2421875 disk.rbd0.out‚0! disk.dm-0.out‚61975.923828125 disk.sda1.out‚0* disk./.usagePercent‚39.469722992474935" disk.nvme0n1.in‚11498.18359375 disk.rbd0.in‚0 cpu‚25.532943706895143 proc.cpu.threads‚17 network.sent‚ 6.776329e+06 disk.sda3.out‚ 379.6640625 disk.dm-0.in‚11498.18359375 disk.sda1.in‚0( proc.memory.availableMB‚ 1894.3984375 disk.drbd2165.in‚0GɘãŸÜ‚(:” ÍêÒ…¯•­ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k”Êê=Ý‚(:² ÍêÒÕó߯ network.recv‚ 1.542093e+06 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1894.7578125* disk./.usagePercent‚39.410815146831176 disk.sda.in‚ 0.2421875 disk.sda.out‚ 380.19140625" disk.nvme0n1.in‚11640.83984375& disk.nvme0n1.out‚61998.86376953125$ memory_percent‚88.43531608581543 proc.cpu.threads‚17# disk./.usageGB‚587.249626159668 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2890.48046875 disk.dm-0.in‚11640.83984375 disk.sda1.out‚0 cpu‚25.533471429531918 disk.sda2.out‚0 disk.rbd2.out‚39.60888671875 disk.sda1.in‚0 disk.dm-1.in‚ 0.2421875 disk.sda3.in‚ 0.2421875( disk.nvme0c0n1.out‚61998.86376953125) proc.memory.percent‚17.64209270477295 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0 disk.rbd2.in‚0.015625# disk.dm-0.out‚61998.86376953125 disk.dm-1.out‚ 380.19140625$ disk.nvme0c0n1.in‚11640.83984375# disk.rbd1.out‚17768.67724609375 disk.sda3.out‚ 380.19140625 network.sent‚ 6.779988e+06 disk.rbd0.in‚0zŠ­ó”Þ‚(:‰ ŸÍêÒÊù¨¬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚223.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.7656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(}y.8ß‚(:­ ŸÍêÒ…ú¯ disk.sda1.in‚0$ disk.nvme0n1.out‚62045.087890625$ memory_percent‚88.43646049499512( proc.memory.availableMB‚ 1894.5703125 disk.rbd2.out‚39.60888671875! disk.dm-0.out‚62045.087890625 disk.drbd2165.out‚0 disk.sda3.out‚ 381.23046875$ disk.nvme0c0n1.in‚11640.83984375& disk.nvme0c0n1.out‚62045.087890625 cpu‚25.533847177629543 proc.cpu.threads‚17* disk./.usagePercent‚39.410815658846246$ disk./.usageGB‚587.2496337890625# disk.rbd1.out‚17768.67724609375 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.2421875 disk.sda.out‚ 381.23046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 381.23046875 disk.sda.in‚ 0.2421875" disk.nvme0n1.in‚11640.83984375# proc.memory.rssMB‚ 2890.48046875 disk.drbd2165.in‚0) proc.memory.percent‚17.64209270477295 disk.sda2.out‚0 disk.dm-0.in‚11640.83984375 disk.sda3.in‚ 0.2421875 network.sent‚ 6.785142e+06 network.recv‚ 1.545853e+06 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875ž÷¸”à‚(:‰ ®ÍêÒ±Èò¬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K•iÂ3á‚(:¨ ®ÍêÒëÁÙ®" disk.dm-0.out‚62058.4072265625 proc.cpu.threads‚17$ disk./.usageGB‚587.2496948242188 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 11641.796875) proc.memory.percent‚17.64209270477295 disk.dm-1.in‚ 0.2421875 disk.sda.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0$ memory_percent‚88.43677043914795) proc.memory.availableMB‚ 1894.51953125 cpu‚25.534301580712828* disk./.usagePercent‚39.410819754966795 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚62058.4072265625 disk.nvme0n1.in‚ 11641.796875# disk.rbd1.out‚17768.73974609375 disk.dm-0.in‚ 11641.796875 disk.rbd2.out‚ 39.65234375# proc.memory.rssMB‚ 2890.48046875 disk.sda.out‚ 381.44921875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 6.790264e+06 network.recv‚ 1.549171e+06 disk.dm-1.out‚ 381.44921875 disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.2421875 disk.sda3.out‚ 381.44921875% disk.nvme0n1.out‚62058.4072265625 disk.sda2.in‚0 disk.sda2.out‚0ØîÜ2â- _wandbruntime‚6070  _runtime‚6070¦ô‘üÈã _step‚297 ' lr-AdamW/pg1‚0.00025590098428778386 ' lr-AdamW/pg2‚0.00025590098428778386 trainer/global_step‚3649 $ _timestamp‚1.784325816844858e+09 _runtime‚6070.219523513¥º­ÍäÇ trainer/global_step‚3649 $ _timestamp‚1.784325816844858e+09 _runtime‚6070.219523513 _step‚297 ' lr-AdamW/pg1‚0.00025590098428778386 ' lr-AdamW/pg2‚0.00025590098428778386©#/bÆ2å- _wandbruntime‚6070  _runtime‚6070DbŒHý'æ÷O ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1049242000064512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3434286963200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1052674915464704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3427066219334 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1052676287027712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31144325 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283450149613568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚6070.220833481 _step‚298 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19747559 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3428410819656 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1049241973891072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4430246790656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1049241973891072 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9366416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287880396404224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1052676287027712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751358 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62761337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10381393 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3434286963200 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31607039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10381148 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843258168460696e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1049242000064512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9366411 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1052674915464704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3427066219334 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1052676287027712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31144325 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283450149613568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚6070.220833481 _step‚298 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19747559 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3428410819656 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1049241973891072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4430246790656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1049241973891072 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9366416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287880396404224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1052676287027712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751358 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62761337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10381393 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3434286963200 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31607039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10381148 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843258168460696e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1049242000064512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9366411 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31554625 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9493561 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4490361468416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚6150.723176584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3479257504256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IáviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31554625 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063142010396672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32042764 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31554619 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587410 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1066155551381082 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32042764 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291815234093568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1062682248048640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1066621267900928 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1066621267900928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3480602893824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1066622639463936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20012154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287324838088704 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9493566 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3474647932764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287324469997568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995çnd(ÿûO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3473303332442 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063142036570112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3479257504256 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063142036570112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32032791 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32032791 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20011900 $ _timestamp‚1.784325897348394e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31554619 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3480602893824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1062682273476608 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1066621267900928 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1066621267900928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3480602893824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1066622639463936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20012154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287324838088704 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9493566 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3474647932764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287324469997568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597389 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1066156921409372 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4490396004864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1066622639463936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1063142010396672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31554625 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9493561 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4490361468416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚6150.723176584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3479257504256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1066155551381082 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32042764 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291815234093568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1062682248048640 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1066435726653906 trainer/global_step‚3699 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31562831 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604094 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9496104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291893521264128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287402324496384 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3481527936512 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32041269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31562825 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063420037300224 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063420011126784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3474226798688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604094 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4491564858880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3475571399122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614087 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31562825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287401956405248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20017218 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9496104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291893521264128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287402324496384 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3481527936512 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32041269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31562825 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063420037300224 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063420011126784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3474226798688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604094 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4491564858880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3475571399122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614087 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31562825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287401956405248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20017218 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1066435726653906 trainer/global_step‚3699 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¯‹K82ƒ- _wandbruntime‚6152  _runtime‚6152òúíó„ÿ /loss/FBt2/detection_loss‚0.7360472049978044 0loss/FRt2/assignment_loss‚1.8195812428238416 1loss/SRqqt1/assignment_loss‚0.531542887293186 .loss/FRt1/detection_loss‚1.429481836851029 /loss/FRt2/detection_loss‚0.6344834639374423 1loss/SRqqt1/detection_loss‚0.6209147483174527 trainer/global_step‚3699 _step‚304 1loss/FBt2/assignment_loss‚0.09953223578945646 /loss/FBt1/detection_loss‚0.6038101270779684 %loss/total_loss‚8.806175653162468 epoch‚9 _runtime‚6152.202676693 0loss/FRt1/assignment_loss‚0.6803172344319961 2loss/SRqqt2/assignment_loss‚0.6788214126384625 1loss/FBt1/assignment_loss‚0.02909248810661304 % _timestamp‚1.7843258988280933e+09 1loss/SRqqt2/detection_loss‚0.9425507708972155M)?¼ …„ .loss/FRt1/detection_loss‚1.429481836851029 /loss/FRt2/detection_loss‚0.6344834639374423 1loss/SRqqt1/detection_loss‚0.6209147483174527 trainer/global_step‚3699 _step‚304 1loss/FBt2/assignment_loss‚0.09953223578945646 /loss/FBt1/detection_loss‚0.6038101270779684 %loss/total_loss‚8.806175653162468 epoch‚9 _runtime‚6152.202676693 0loss/FRt1/assignment_loss‚0.6803172344319961 2loss/SRqqt2/assignment_loss‚0.6788214126384625 1loss/FBt1/assignment_loss‚0.02909248810661304 % _timestamp‚1.7843258988280933e+09 1loss/SRqqt2/detection_loss‚0.9425507708972155 /loss/FBt2/detection_loss‚0.7360472049978044 0loss/FRt2/assignment_loss‚1.8195812428238416 1loss/SRqqt1/assignment_loss‚0.531542887293186°oõ Ÿ†‚(:” —ÎêÒŒ‡¬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NâGŽ1‡‚(:¦ —ÎêÒÑ‘°¯$ disk./.usageGB‚587.2546310424805! disk.nvme0n1.in‚ 12221.9765625 disk.rbd0.in‚0 disk.dm-0.in‚ 12221.9765625 disk.rbd2.in‚0.015625 network.sent‚ 6.884076e+06 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 12221.9765625% disk.nvme0n1.out‚62141.3173828125 disk.sda.out‚ 389.4140625 network.recv‚ 1.574281e+06 disk.sda1.in‚0 disk.sda3.out‚ 389.4140625 disk.sda2.out‚0# proc.memory.rssMB‚ 2890.48046875 disk.drbd2165.in‚0 disk.rbd2.out‚ 39.7783203125" disk.dm-0.out‚62141.3173828125 disk.sda.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚62141.3173828125 disk.sda2.in‚0$ memory_percent‚88.43874931335449( proc.memory.availableMB‚ 1894.1953125) proc.memory.percent‚17.64209270477295 proc.cpu.threads‚17 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 389.4140625 disk.sda3.in‚ 0.2421875# disk.rbd1.out‚17768.94287109375( disk./.usagePercent‚39.4111510287161 cpu‚25.53755088401947ˆ «Ïˆ‚(:’ ¦ÎêÒ÷’Ŭ gpu.0.gpu‚81.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±‡Âž6‰‚(:« ¦ÎêÒ‹ºà® disk.dm-0.in‚ 12222.4140625$ memory_percent‚88.44046592712402 proc.cpu.threads‚17# disk.dm-0.out‚62153.67431640625# proc.memory.rssMB‚ 2890.48046875 network.sent‚ 6.947736e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.2421875 disk.rbd0.out‚0 disk.rbd2.in‚0.015625( proc.memory.availableMB‚ 1893.9140625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 12222.4140625 disk.sda2.out‚0# disk.rbd1.out‚17769.00537109375 cpu‚25.538150357970213 disk.sda3.out‚ 391.4609375 disk.sda1.out‚0! disk.nvme0n1.in‚ 12222.4140625& disk.nvme0n1.out‚62153.67431640625 disk.sda2.in‚0) proc.memory.percent‚17.64209270477295 network.recv‚ 1.579163e+06) disk./.usagePercent‚39.41092753413871 disk.sda.in‚ 0.2421875 disk.rbd1.in‚ 506.4921875$ disk./.usageGB‚587.2513008117676 disk.dm-1.in‚ 0.2421875 disk.dm-1.out‚ 391.4609375 disk.sda.out‚ 391.4609375 disk.rbd0.in‚0 disk.rbd2.out‚ 39.7783203125( disk.nvme0c0n1.out‚62153.67431640625 disk.drbd2165.out‚0I¾'D”Š‚(:‰ µÎêÒ©ªÑ« gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zej5‹‚(:ª µÎêÒýÁ·® disk.rbd2.in‚0.015625 disk.rbd2.out‚ 39.7783203125 disk.dm-1.out‚ 392.25390625& disk.nvme0c0n1.out‚62169.228515625 disk.sda2.in‚0$ memory_percent‚88.43903541564941# proc.memory.rssMB‚ 2890.48046875 disk.rbd0.out‚0 network.recv‚ 1.582564e+06 disk.dm-0.in‚12223.37109375 disk.sda3.in‚ 0.2421875$ disk.nvme0c0n1.in‚12223.37109375 disk.sda.out‚ 392.25390625 network.sent‚ 6.952304e+06$ disk./.usageGB‚587.2513999938965 disk.dm-1.in‚ 0.2421875$ disk.nvme0n1.out‚62169.228515625 disk.rbd1.in‚ 506.4921875) proc.memory.percent‚17.64209270477295 proc.cpu.threads‚17 disk.drbd2165.out‚0) disk./.usagePercent‚39.41093419033459! disk.dm-0.out‚62169.228515625 disk.drbd2165.in‚0 disk.sda3.out‚ 392.25390625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.2421875# disk.rbd1.out‚17769.00537109375( proc.memory.availableMB‚ 1894.1484375 cpu‚25.53858363055494 disk.sda1.out‚0" disk.nvme0n1.in‚12223.37109375 disk.sda2.out‚0„bg<Œ‚(:’ ÄÎêÒž¢º¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bt7ô6‚(:« ÄÎêÒ£Äí® disk.dm-1.out‚ 392.5546875 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 39.7783203125 disk.dm-0.in‚12308.18359375 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.64209270477295 network.recv‚ 1.586194e+06& disk.nvme0n1.out‚62173.32080078125 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 network.sent‚ 6.957901e+06 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.539008472766717 proc.cpu.threads‚17( disk.nvme0c0n1.out‚62173.32080078125$ disk./.usageGB‚587.2513961791992# disk.dm-0.out‚62173.32080078125 disk.dm-1.in‚ 0.2421875& proc.memory.availableMB‚ 1894.15625) disk./.usagePercent‚39.41093393432706 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.2421875# memory_percent‚88.4389877319336# proc.memory.rssMB‚ 2890.48046875 disk.sda.in‚ 0.2421875 disk.sda.out‚ 392.5546875 disk.sda3.out‚ 392.5546875$ disk.nvme0c0n1.in‚12308.18359375" disk.nvme0n1.in‚12308.18359375# disk.rbd1.out‚17769.06787109375jغܟŽ‚(:” ÓÎêÒô×ð« gpu.0.gpu‚83.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚132.419' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.483800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÿ¾+‚(:  ÓÎêÒáðÒ® disk.drbd2165.in‚0 disk.dm-1.out‚394.5625 network.sent‚ 6.962427e+06* disk./.usagePercent‚39.410841771614734# disk.nvme0c0n1.in‚ 12398.2421875' disk.nvme0c0n1.out‚62197.6357421875! disk.nvme0n1.in‚ 12398.2421875 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0" disk.dm-0.out‚62197.6357421875$ disk./.usageGB‚587.2500228881836 disk.sda3.out‚394.5625 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1894.1484375 cpu‚25.539596614887603 disk.sda3.in‚ 0.2421875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.2421875$ memory_percent‚88.43903541564941# proc.memory.rssMB‚ 2890.48046875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.2421875# disk.rbd1.out‚17769.06787109375 disk.rbd2.out‚ 39.7783203125% disk.nvme0n1.out‚62197.6357421875 disk.sda2.in‚0 disk.rbd2.in‚0.015625 network.recv‚ 1.589787e+06 disk.sda.out‚394.5625 disk.dm-0.in‚ 12398.2421875) proc.memory.percent‚17.64209270477295«Mçk2- _wandbruntime‚6231  _runtime‚6231¿q°É‘à ' lr-AdamW/pg1‚0.00025479293212451635 ' lr-AdamW/pg2‚0.00025479293212451635 trainer/global_step‚3749 % _timestamp‚1.7843259779078467e+09 _runtime‚6231.282456498 _step‚305šó,ŠÎ’È ' lr-AdamW/pg1‚0.00025479293212451635 ' lr-AdamW/pg2‚0.00025479293212451635 trainer/global_step‚3749 % _timestamp‚1.7843259779078467e+09 _runtime‚6231.282456498 _step‚305±ß:~!2“- _wandbruntime‚6231  _runtime‚6231¡V‡âý'”÷O RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9620711 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 trainer/global_step‚3749 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64423462 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1077042046902272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3519540445550 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526918824448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291199158472704 _step‚306 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295749770264064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚6231.283640613 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32468516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 % _timestamp‚1.7843259779088855e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10656381 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32458543 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1080568991900160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32458543 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295749367636480 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1080568991900160 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1080567620337152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4550611791360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525573434880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚016è•üO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525573434880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9620711 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 trainer/global_step‚3749 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64423462 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1077042046902272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3519540445550 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526918824448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291199158472704 _step‚306 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inaþ6àh ctive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295749770264064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚6231.283640613 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32468516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 % _timestamp‚1.7843259779088855e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10656381 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32458543 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1080568991900160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32458543 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295749367636480 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1080568991900160 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1080567620337152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4550611791360²3¿uÈ2–- _wandbruntime‚6232  _runtime‚6232ãK>'—€N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3520463911796 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291276644880384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1080847917672960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚307 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9623259 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1080846546102784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450139 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20282358 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1077320047632384 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3527843867136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31973125 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1077320073805824 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1076854244163584 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1076854218735616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784325979305292e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9623254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295828449815040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1077320073805824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1080847917672960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295828047194624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450139 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4551804934656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1080376052675814 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32467021 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526498470400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31973125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3521808512230 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1077320047632384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4551770405376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32477008 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450139 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20282358 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1077320047632384 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3527843867136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31973125 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1077320073805824 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1076854244163584 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1076854218735616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784325979305292e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9623254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295828449815040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1077320073805824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1080847917672960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295828047194624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450139 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4551804934656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1080376052675814 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32467021 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526498470400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31973125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3521808512230 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1077320047632384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4551770405376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32477008 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416³Öøs2™- _wandbruntime‚6232  _runtime‚6232á> š„ 3loss/FBt2/assignment_loss‚0.0010199524320200297 /loss/FRt1/detection_loss‚1.4555968546071354 /loss/FBt1/detection_loss‚0.5470275523349348 %loss/total_loss‚8.668554077518854 /loss/FRt2/detection_loss‚0.5783946813068082 1loss/SRqqt2/detection_loss‚0.9197541491170838 /loss/FBt2/detection_loss‚0.8470912317710894 epoch‚9 trainer/global_step‚3749 0loss/FRt1/assignment_loss‚0.7333120374747042 0loss/FRt2/assignment_loss‚1.8598943189422101 3loss/SRqqt1/assignment_loss‚0.49745991091530367 0loss/SRqqt1/detection_loss‚0.653886798206902 % _timestamp‚1.7843259793830588e+09 _runtime‚6232.757690117 _step‚308 2loss/SRqqt2/assignment_loss‚0.5441004283831233 2loss/FBt1/assignment_loss‚0.031016162027538575˜oÆ›‰ /loss/FBt1/detection_loss‚0.5470275523349348 %loss/total_loss‚8.668554077518854 /loss/FRt2/detection_loss‚0.5783946813068082 1loss/SRqqt2/detection_loss‚0.9197541491170838 /loss/FBt2/detection_loss‚0.8470912317710894 epoch‚9 trainer/global_step‚3749 0loss/FRt1/assignment_loss‚0.7333120374747042 0loss/FRt2/assignment_loss‚1.8598943189422101 3loss/SRqqt1/assignment_loss‚0.49745991091530367 0loss/SRqqt1/detection_loss‚0.653886798206902 % _timestamp‚1.7843259793830588e+09 _runtime‚6232.757690117 _step‚308 2loss/SRqqt2/assignment_loss‚0.5441004283831233 2loss/FBt1/assignment_loss‚0.031016162027538575 3loss/FBt2/assignment_loss‚0.0010199524320200297 /loss/FRt1/detection_loss‚1.4555968546071354´d‹S’œ‚(:‡ âÎêÒþ¨Î­ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚242.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aá :‚(:¯ âÎêÒ‰”¯ disk.dm-0.in‚12399.19921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.2421875) proc.memory.availableMB‚ 1893.41796875 disk.rbd2.in‚0.015625$ disk.nvme0c0n1.in‚12399.19921875( disk.nvme0c0n1.out‚62210.82080078125 disk.sda.in‚ 0.2421875 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚62210.82080078125) disk./.usagePercent‚39.41064822991885 disk.sda3.in‚ 0.2421875 disk.sda3.out‚ 397.1328125" disk.nvme0n1.in‚12399.19921875 disk.sda.out‚ 397.1328125 disk.rbd2.out‚ 39.7783203125# disk.rbd1.out‚17769.13037109375$ memory_percent‚88.44349384307861 disk.drbd2165.out‚0 disk.dm-1.out‚ 397.1328125 network.sent‚ 6.968078e+06 network.recv‚ 1.593371e+06& disk.nvme0n1.out‚62210.82080078125) proc.memory.percent‚17.64209270477295 cpu‚25.539936410716418 disk.drbd2165.in‚0$ disk./.usageGB‚587.2471389770508 disk.sda2.out‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2890.48046875t>唞‚(:‰ ñÎêÒþžÂ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†˜‰,Ÿ‚(:¡ ñÎêÒµ°° disk.dm-0.out‚62217.35546875 disk.sda1.out‚0 disk.sda3.out‚ 397.8046875# disk.rbd1.out‚17769.13037109375 network.sent‚ 7.030763e+06$ disk.nvme0c0n1.in‚12399.19921875" disk.nvme0n1.in‚12399.19921875# disk.nvme0n1.out‚62217.35546875 disk.sda.in‚ 0.2421875) disk./.usagePercent‚39.41065130200926# disk./.usageGB‚587.247184753418 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 397.8046875 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 cpu‚25.5403881638651 network.recv‚ 1.598633e+06% disk.nvme0c0n1.out‚62217.35546875 disk.sda.out‚ 397.8046875 disk.rbd0.out‚0 disk.rbd2.out‚ 39.7783203125 disk.dm-0.in‚12399.19921875 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚88.43927383422852# proc.memory.rssMB‚ 2890.48046875 disk.sda3.in‚ 0.2421875 disk.sda2.out‚0 disk.rbd2.in‚0.015625' proc.memory.availableMB‚ 1894.109375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.2421875) proc.memory.percent‚17.64209270477295F®v• ‚(:Š €ÏêÒ̘ÿ« gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï¿ñ¥)¡‚(:ž €ÏêÒªŸ«® disk.sda2.out‚0 disk.sda3.out‚ 398.40625 disk.rbd1.in‚ 506.4921875 disk.rbd2.out‚ 39.7783203125 disk.sda1.in‚0 network.sent‚ 7.035867e+06 disk.sda.out‚ 398.40625 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 398.40625 disk.nvme0n1.in‚ 12400.03125 disk.sda.in‚ 0.2421875# disk.rbd1.out‚17769.19287109375# disk.dm-0.out‚62225.71533203125 disk.sda1.out‚0$ memory_percent‚88.44079971313477 disk.dm-1.in‚ 0.2421875! disk.nvme0c0n1.in‚ 12400.03125( disk.nvme0c0n1.out‚62225.71533203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 12400.03125 proc.cpu.threads‚17# proc.memory.rssMB‚ 2890.48046875$ disk./.usageGB‚587.2481536865234 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚62225.71533203125 disk.sda2.in‚0) proc.memory.percent‚17.64209270477295 network.recv‚ 1.601972e+06' proc.memory.availableMB‚ 1893.859375 cpu‚25.54098426067029* disk./.usagePercent‚39.410716327922955 disk.sda3.in‚ 0.2421875Ғ¢‚(:‡ ÏêÒ€õŽ¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Œã”8£‚(:­ ÏêÒžçÆ® disk.rbd1.in‚ 506.4921875# disk.rbd1.out‚17769.19287109375$ disk.nvme0c0n1.in‚12484.27734375 disk.sda2.out‚0) proc.memory.percent‚17.64209270477295 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.2421875 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 39.7783203125" disk.dm-0.out‚62253.2412109375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚12484.27734375# proc.memory.rssMB‚ 2890.48046875 disk.sda1.out‚0" disk.nvme0n1.in‚12484.27734375 disk.sda.out‚ 400.86328125 disk.rbd0.in‚0( proc.memory.availableMB‚ 1893.8515625 network.sent‚ 7.04145e+06$ disk./.usageGB‚587.2480621337891 disk.dm-1.out‚ 400.86328125 disk.sda3.in‚ 0.2421875% disk.nvme0n1.out‚62253.2412109375 disk.sda.in‚ 0.2421875 cpu‚25.541241568484097 disk.sda1.in‚0$ memory_percent‚88.44084739685059 proc.cpu.threads‚17 network.recv‚ 1.605643e+06) disk./.usagePercent‚39.41071018374214 disk.drbd2165.out‚0 disk.sda3.out‚ 400.86328125' disk.nvme0c0n1.out‚62253.2412109375õƽ’¤‚(:‡ žÏêÒù€Ä¬ gpu.0.gpu‚31.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´{äå<¥‚(:± žÏêÒÃÁ—¯ disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚62268.07958984375$ memory_percent‚88.44201564788818 network.recv‚ 1.609373e+06 cpu‚25.54181037289431 disk.sda3.in‚ 0.2421875 disk.dm-1.out‚ 402.96484375$ disk./.usageGB‚587.2524528503418$ disk.nvme0c0n1.in‚12484.27734375" disk.nvme0n1.in‚12484.27734375& disk.nvme0n1.out‚62268.07958984375 disk.sda.out‚ 402.96484375# proc.memory.rssMB‚ 2890.48046875 network.sent‚ 7.046596e+06 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 39.7783203125) proc.memory.availableMB‚ 1893.66015625) proc.memory.percent‚17.64209270477295 proc.cpu.threads‚17( disk.nvme0c0n1.out‚62268.07958984375 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875 disk.drbd2165.in‚0 disk.sda3.out‚ 402.96484375 disk.rbd2.in‚0.015625 disk.dm-0.in‚12484.27734375 disk.dm-1.in‚ 0.2421875) disk./.usagePercent‚39.41100484841405 disk.sda.in‚ 0.2421875# disk.rbd1.out‚17769.25537109375 disk.rbd0.out‚0]§^2¦- _wandbruntime‚6310  _runtime‚6310öCçɧà _runtime‚6310.814391701 _step‚309 ' lr-AdamW/pg1‚0.00025422754289798106 ' lr-AdamW/pg2‚0.00025422754289798106 trainer/global_step‚3799 % _timestamp‚1.7843260574397438e+09àIÜ&Î¨È _runtime‚6310.814391701 _step‚309 ' lr-AdamW/pg1‚0.00025422754289798106 ' lr-AdamW/pg2‚0.00025422754289798106 trainer/global_step‚3799 % _timestamp‚1.7843260574397438e+09µ¸ ‘å2©- _wandbruntime‚6310  _runtime‚6310HµWáªöO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3573234755072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295073110765568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1094515344336384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10794380 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bI¼”;%ytes.large_pool.allocated‚295073478856704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1090942109581312 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1090942083407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20542241 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299684213741056 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259514 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3571889365504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9747861 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784326057440781e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32375219 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3565777558658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32894268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4610700347904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9747866 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32375225 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1090942109581312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32894268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1094036203424898 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094513972773376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3571889365504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10794631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32884295 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32375225 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32884295 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1094515344336384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3573234755072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269493 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20542497 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094513972773376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3571889365504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10794631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32884295 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32375225 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32884295 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1094515344336384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3573234755072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269493 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20542497 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1094515344336384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10794380 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295073478856704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1090942109581312 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1090942083407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20542241 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299684213741056 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259514 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3571889365504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9747861 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784326057440781e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32375219 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3565777558658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820¶„…ú2¬- _wandbruntime‚6312  _runtime‚6312å_º'­N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295150597173248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚3799 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3568045625338 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1091220110311424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276198 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20547526 _runtime‚6312.203995289 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094794270109184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32902760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843260588291297e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1094792898539008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286191 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20547771 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3574159797760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10797362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1094316378697722 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276198 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32383431 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299762894774784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4611894981120 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1090748307644416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295150597173248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚3799 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3568045625338 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1091220110311424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276198 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20547526 _runtime‚6312.203995289 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094794270109184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32902760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843260588291297e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1094792898539008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286191 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20547771 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3574159797760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10797362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1094316378697722 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276198 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32383431 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299762894774784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4611894981120 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1090748307644416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _step‚314 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9875016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095566 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1104842119913472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1104842146086912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843261365528579e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32785519 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1108461696772608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618205296128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303618508699136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4670709458432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1104842119913472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33320020 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1107977899475096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3612014671766 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1104364540203008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1107976529446806 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1108460325209600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3619550685696 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4670674921984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108460325209600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303618106071552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20807856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3619550685696 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32785525 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33320020 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298947799240704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298947431149568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32785519 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20808112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32785525 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618205296128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10933096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10932845 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119¾ î»ÁüO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1108461696772608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1104842146086912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9875011 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33310047 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33310047 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _step‚314 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9875016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095566 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1104842119913472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1104842146086912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843261365528579e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32785519 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1108461696772608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618205296128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303618508699136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4670709458432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1104842119913472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33320020 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1107977899475096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3612014671766 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1104364540203008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1107976529446806 JBDeviceStatsMonitor.on_Jg(Gtrain_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1108460325209600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3619550685696 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4670674921984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108460325209600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303618106071552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20807856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3619550685696 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32785525 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33320020 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298947799240704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298947431149568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32785519 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20808112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32785525 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618205296128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10933096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10932845º KŽáŸÂ‚(:” øÏêÒËÿ¬¬ gpu.0.gpu‚85.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚293.169' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.633799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xåf@2Â(:§ øÏêÒ€çᯠdisk.sda1.in‚0 network.sent‚ 7.134291e+06 network.recv‚ 1.631191e+06$ disk./.usageGB‚587.2455139160156 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚12541.67578125 proc.cpu.threads‚17" disk.nvme0n1.in‚12541.67578125 disk.sda2.out‚0$ memory_percent‚88.44435214996338 disk.sda.out‚ 410.47265625 disk.dm-1.in‚ 0.8359375! proc.memory.rssMB‚ 2890.484375) proc.memory.percent‚17.64211654663086# disk.nvme0n1.out‚62347.16796875 disk.sda1.out‚0) proc.memory.availableMB‚ 1893.27734375 cpu‚25.544453753486216* disk./.usagePercent‚39.410539170709264 disk.sda.in‚ 0.8359375 disk.rbd2.out‚ 40.623046875 disk.sda2.in‚0 disk.rbd1.in‚ 506.4921875 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 410.47265625# disk.rbd1.out‚17769.44287109375 disk.dm-0.out‚62347.16796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 410.47265625% disk.nvme0c0n1.out‚62347.16796875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚12541.67578125²ô2Ä- _wandbruntime‚6391  _runtime‚6391›•œò'ÅN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9877559 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3614282738446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32793725 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33318525 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1108255334691228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1105120146817024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104642396553216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20813143 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122243 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299024917557248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33328512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4671879352832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33328512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9877554 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32793725 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10935589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303697197433344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640•» 'ƆN PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9877559 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3614282738446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32793725 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33318525 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1108255334691228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1105120146817024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104642396553216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20813143 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122243 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299024917557248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33328512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4671879352832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33328512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9877554 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32793725 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10935589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303697197433344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6»lÅmË2Ç- _wandbruntime‚6391  _runtime‚6391ëâ/È€ 2loss/FBt1/assignment_loss‚0.012044994762368299 0loss/SRqqt1/detection_loss‚0.653869993551699 /loss/FBt2/detection_loss‚1.0620804831166477 $loss/total_loss‚8.96717164907783 epoch‚9 _runtime‚6391.401257323 2loss/SRqqt1/assignment_loss‚0.4808084983433983 /loss/FRt2/detection_loss‚0.5718089032910287 trainer/global_step‚3849 0loss/FRt1/assignment_loss‚0.7607133628978123 0loss/FRt2/assignment_loss‚1.8278109634637598 1loss/SRqqt2/detection_loss‚0.9762209603111214 $ _timestamp‚1.784326138026899e+09 _step‚316 2loss/SRqqt2/assignment_loss‚0.6851911578902291 2loss/FBt2/assignment_loss‚0.042581177798330615 /loss/FRt1/detection_loss‚1.3878838050106317 /loss/FBt1/detection_loss‚0.5061573486408039Iû{ É… epoch‚9 _runtime‚6391.401257323 2loss/SRqqt1/assignment_loss‚0.4808084983433983 /loss/FRt2/detection_loss‚0.5718089032910287 trainer/global_step‚3849 0loss/FRt1/assignment_loss‚0.7607133628978123 0loss/FRt2/assignment_loss‚1.8278109634637598 1loss/SRqqt2/detection_loss‚0.9762209603111214 $ _timestamp‚1.784326138026899e+09 _step‚316 2loss/SRqqt2/assignment_loss‚0.6851911578902291 2loss/FBt2/assignment_loss‚0.042581177798330615 /loss/FRt1/detection_loss‚1.3878838050106317 /loss/FBt1/detection_loss‚0.5061573486408039 2loss/FBt1/assignment_loss‚0.012044994762368299 0loss/SRqqt1/detection_loss‚0.653869993551699 /loss/FBt2/detection_loss‚1.0620804831166477 $loss/total_loss‚8.96717164907783¼ž®e,’Ê‚(:‡ ‡ÐêÒ‹†è­ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÒ\³<Ë‚(:± ‡ÐêÒº¡§® disk.sda3.in‚ 0.8359375$ memory_percent‚88.44478130340576! proc.memory.rssMB‚ 2890.484375 cpu‚25.544828639289967 network.sent‚ 7.186923e+06 network.recv‚ 1.636004e+06 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.dm-1.out‚ 411.50390625" disk.nvme0n1.in‚12673.28515625# disk.dm-0.out‚62395.68896484375) proc.memory.percent‚17.64211654663086 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0& disk.nvme0n1.out‚62395.68896484375) proc.memory.availableMB‚ 1893.20703125* disk./.usagePercent‚39.201581469134254# disk./.usageGB‚584.131893157959 disk.sda1.out‚0$ disk.nvme0c0n1.in‚12673.28515625 disk.sda.in‚ 0.8359375 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda.out‚ 411.50390625 disk.sda2.in‚0# disk.rbd1.out‚17769.44287109375 disk.rbd2.out‚40.74169921875 disk.sda1.in‚0 disk.sda3.out‚ 411.50390625( disk.nvme0c0n1.out‚62395.68896484375 proc.cpu.threads‚17 disk.dm-0.in‚12673.28515625À᚟̂(:” –ÐêÒÀâØ« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚301.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.268600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0щR.Í‚(:£ –ÐêÒȆë­ disk.sda3.out‚ 412.1484375# disk.rbd1.out‚17769.50537109375 disk.drbd2165.in‚0 disk.rbd2.in‚0.015625 cpu‚25.545154897918042 proc.cpu.threads‚17 network.recv‚ 1.640892e+06* disk./.usagePercent‚39.201591453428094 disk.dm-1.out‚ 412.1484375) proc.memory.availableMB‚ 1893.23828125 disk.sda1.out‚0" disk.nvme0n1.in‚12673.28515625" disk.nvme0n1.out‚ 62399.3984375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375 disk.sda.in‚ 0.8359375 disk.sda.out‚ 412.1484375 disk.rbd2.out‚40.74169921875! proc.memory.rssMB‚ 2890.484375) proc.memory.percent‚17.64211654663086 network.sent‚ 7.238889e+06 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-0.in‚12673.28515625 disk.dm-0.out‚ 62399.3984375$ memory_percent‚88.44459056854248$ disk.nvme0c0n1.out‚ 62399.3984375$ disk./.usageGB‚584.1320419311523$ disk.nvme0c0n1.in‚12673.28515625 disk.rbd1.in‚ 506.4921875 disk.rbd0.out‚0 disk.sda1.in‚0ÿöŸÎ‚(:” ¥ÐêÒ¿×ß­ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«j ª,Ï‚(:¡ ¥Ðêҿ裱 disk.dm-0.in‚ 12674.2421875 disk.sda1.out‚0 disk.dm-1.in‚ 0.8359375) proc.memory.percent‚17.64211654663086 disk.rbd0.out‚0 disk.sda3.out‚ 412.515625 network.sent‚ 7.244495e+06 disk.rbd2.in‚0.015625! disk.nvme0n1.in‚ 12674.2421875 cpu‚25.54564101205486 network.recv‚ 1.644332e+06 disk.rbd2.out‚ 40.78515625( proc.memory.availableMB‚ 1893.6953125" disk.dm-0.out‚62409.4853515625' disk.nvme0c0n1.out‚62409.4853515625 disk.sda2.in‚0$ memory_percent‚88.44180107116699 disk.drbd2165.out‚0 disk.sda.out‚ 412.515625 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375! proc.memory.rssMB‚ 2890.484375$ disk./.usageGB‚584.1321258544922 disk.drbd2165.in‚0 disk.dm-1.out‚ 412.515625# disk.nvme0c0n1.in‚ 12674.2421875% disk.nvme0n1.out‚62409.4853515625 disk.sda.in‚ 0.8359375# disk.rbd1.out‚17769.50537109375 proc.cpu.threads‚17* disk./.usagePercent‚39.201597085593846å*ÄЂ(:’ ´ÐêÒž ¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>¡üæÑ‚(:¬ ´ÐêÒ΄½® disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 412.76171875 disk.sda1.out‚0( proc.memory.availableMB‚ 1893.6484375 disk.sda2.out‚0 network.sent‚ 7.249057e+06$ disk./.usaÑ×OQgeGB‚584.1320495605469$ disk.nvme0c0n1.in‚12675.07421875" disk.nvme0n1.in‚12675.07421875 disk.rbd2.in‚0.015625 disk.rbd0.in‚0 disk.dm-0.in‚12675.07421875 disk.drbd2165.out‚0$ memory_percent‚88.44208717346191 disk.sda.in‚ 0.8359375 network.recv‚ 1.647473e+06( disk.nvme0c0n1.out‚62417.19482421875 disk.rbd1.in‚ 506.4921875 disk.dm-1.in‚ 0.8359375) proc.memory.percent‚17.64211654663086* disk./.usagePercent‚39.201591965443164# disk.dm-0.out‚62417.19482421875 disk.dm-1.out‚ 412.76171875! proc.memory.rssMB‚ 2890.484375 disk.rbd2.out‚ 40.7890625 disk.drbd2165.in‚0& disk.nvme0n1.out‚62417.19482421875# disk.rbd1.out‚17769.63037109375 disk.sda3.out‚ 412.76171875 disk.rbd0.out‚0 cpu‚25.54590565925995 disk.sda2.in‚0 disk.sda3.in‚ 0.8359375úXö”Ò‚(:‰ ÃÐêÒ´§Ç¬ gpu.0.gpu‚41.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚344.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%¨ÖË0Ó‚(:¥ ÃÐêÒâµ® network.sent‚ 7.254762e+06 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 cpu‚25.546493466735193 network.recv‚ 1.651123e+06 disk.sda1.in‚0 disk.dm-1.out‚ 413.05078125 disk.sda3.in‚ 0.8359375$ memory_percent‚88.44563961029053 disk.rbd0.in‚0& disk.nvme0n1.out‚62422.98095703125 disk.sda.out‚ 413.05078125# disk.rbd1.out‚17769.63037109375# disk.dm-0.out‚62422.98095703125 disk.rbd1.in‚ 506.4921875) proc.memory.availableMB‚ 1893.06640625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.out‚ 41.08203125 disk.sda1.out‚0 disk.sda3.out‚ 413.05078125 disk.sda2.in‚0) disk./.usagePercent‚39.20159682958631 disk.dm-0.in‚ 12819.21875 disk.dm-1.in‚ 0.8359375! disk.nvme0c0n1.in‚ 12819.21875 disk.nvme0n1.in‚ 12819.21875) proc.memory.percent‚17.64211654663086$ disk./.usageGB‚584.1321220397949 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚62422.98095703125 disk.sda.in‚ 0.8359375 disk.sda2.out‚0! proc.memory.rssMB‚ 2890.484375Á‹P!2Ô- _wandbruntime‚6469  _runtime‚6469çW±ÉÕà % _timestamp‚1.7843262159117637e+09 _runtime‚6469.286359172 _step‚317 ' lr-AdamW/pg1‚0.00025307421972224257 ' lr-AdamW/pg2‚0.00025307421972224257 trainer/global_step‚3899ÄZdÎÖÈ trainer/global_step‚3899 % _timestamp‚1.7843262159117637e+09 _runtime‚6469.286359172 _step‚317 ' lr-AdamW/pg1‚0.00025307421972224257 ' lr-AdamW/pg2‚0.00025307421972224257½µüÉk2×- _wandbruntime‚6469  _runtime‚6469#,HÏÿ'ØùO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3659596385196 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122408049208832 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118742182592512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122406677645824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33195819 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1118742156419072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33735799 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1121918225497004 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118258603683840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941597 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4731189783040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3665866616320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307552908877312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843262159128554e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3664521226752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10002166 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33745772 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21073746 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚3899 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118742156419072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11071580 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3664521226752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10002161 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚318 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1122408049208832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3658251784874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33735799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3665866616320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307553309407744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚302822119624704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1118258629111808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640\Szr(ÙþO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122406677645824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33195819 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118742182592512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122406677645824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33195819 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1118742156419072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33735799 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1121918225497004 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118258603683840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941597 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4731189783040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3665866616320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307552908877312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843262159128554e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3664521226752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10002166 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33745772 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21073746 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚3899 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118742156419072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11071580 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3664521226752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10002161 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚318 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1122408049208832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3658251784874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33735799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3665866616320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307553309407744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚302822119624704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1118258629111808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3659596385196 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122408049208832 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33204031 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3659175251120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1122686974981632 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307631601870336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118536485462016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1122197030741538 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33204031 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10004704 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958295 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1119020157149184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3666791659008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21078810 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3665446262272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33204025 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33754264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307632002393600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33754264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33744277 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4732363929088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784326217297761e+09 _runtime‚6470.672593342 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚319 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚3899 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302899237941248 D>DeviceStatsMonitor.on_train_bWR[šatch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1122686974981632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11074106 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993zw¬ '܇N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11074334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784326217297761e+09 _runtime‚6470.672593342 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚319 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚3899 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302899237941248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1122686974981632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11074106 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3665446262272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66958295 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1119020183322624 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33204031 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3659175251120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1122686974981632 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307631601870336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118536485462016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1122197030741538 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33204031 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10004704 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958295 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1119020157149184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3666791659008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21078810 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3665446262272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33204025 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33754264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307632002393600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33754264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33744277 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4732363929088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136353030082048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311487672398336 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33606119 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843262952776394e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132642219098112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4791197853184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33606125 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6548.652446829 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21339181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10129311 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1135857181490622 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34171524 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21339440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10129316 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11209870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311487269770752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3710837157376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11210124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34161551 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34171524 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3710837157376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132152692592640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67767670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132152718020608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3705833498304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1136354401645056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3712182546944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306696071917568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1132642219098112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33606125 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1135858551518912 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 13700282909ùñ¯ÁïþO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_staXµCrt/active_bytes.small_pool.freed‚ 3710837157376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11210124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34161551 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34171524 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3710837157376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132152692592640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67767670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132152718020608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3705833498304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1136354401645056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3712182546944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306696071917568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1132642219098112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33606125 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1135858551518912 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136353030082048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311487672398336 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33606119 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843262952776394e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132642219098112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4791197853184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33606125 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6548.652446829 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21339181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10129311 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1135857181490622 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34171524 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21339440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10129316 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11209870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311487269770752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968Âr \2ð- _wandbruntime‚6550  _runtime‚6550xP 'ñƒN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784354 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34180016 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3711762192896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794347 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚3949 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344479 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4792423566848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1132430574370816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843262966633742e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3713107589632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11212625 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6550.038291703 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3713107589632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33614325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10131859 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794347 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _step‚323 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132920219828224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10131854 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1136135986735044 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33614331 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34180016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1136633327417856 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1136633327417856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11212865 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34170029 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3706756964662 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843262966633742e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3713107589632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11212625 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6550.038291703 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3713107589632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33614325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10131859 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794347 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _step‚323 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132920219828224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10131854 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1136135986735044 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33614331 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34180016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1136633327417856 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1136633327417856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11212865 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34170029 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3706756964662 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784354 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34180016 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3711762192896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794347 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚3949 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344479 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4792423566848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1132430574370816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34016419 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚3999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758498477568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1150300754081280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1150300754081280 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1146542255603712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3752070611412 _runtime‚6627.783723071 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34597276 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34587303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4851186917888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1146046781501440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10256461 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1149797507512530 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757153088000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1146542229430272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34597276 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750726011090 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34016419 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3757153088000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34016425 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315421981847040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4851221454336 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1150299382518272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11348535 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315421579219456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚326 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34597276 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34587303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4851186917888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1146046781501440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10256461 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1149797507512530 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757153088000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1146542229430272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34597276 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750726011090 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34016419 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3757153088000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34016425 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315421981847040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4851221454336 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1150299382518272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11348535 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315421579219456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚326 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34016419 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚3999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758498477568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1150300754081280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1150300754081280 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1146542255603712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3752070611412 _runtime‚6627.783723071 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759423520256 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759423520256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21610294 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34024631 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1150077682785354 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21610049 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310648246800384 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758078123520 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843263757968864e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758078123520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4852419340800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11351045 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1146820256333824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146820256333824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315500263520768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3751649477336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315500666141184 trainer/global_step‚3999 _step‚327 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1150579679854080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34605768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.iåÛÜ¥ peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1146820230160384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34024625 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310647878709248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630399 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10259004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1150578308283904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34595781 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620406 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÓŠ=Œ'ˆˆN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10259009 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620406 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752994077770 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759423520256 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759423520256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21610294 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34024631 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1150077682785354 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21610049 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310648246800384 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758078123520 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843263757968864e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758078123520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4852419340800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11351045 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1146820256333824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146820256333824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315500263520768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3751649477336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315500666141184 trainer/global_step‚3999 _step‚327 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1150579679854080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34605768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1146820230160384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34024625 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310647878709248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630399 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10259004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1150578308283904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34595781 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620406 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ç„è?®2‰- _wandbruntime‚6629  _runtime‚6629±æé Š€ trainer/global_step‚3999 0loss/FRt2/assignment_loss‚1.8197753395978902 /loss/FRt2/detection_loss‚0.5826529553830987 /loss/FBt2/detection_loss‚0.6750555617515656 0loss/SRqqt1/detection_loss‚0.682867293599167 %loss/total_loss‚8.864732447417683 epoch‚9 % _timestamp‚1.7843263758820047e+09 _runtime‚6629.256345238 _step‚328 2loss/SRqqt1/assignment_loss‚0.5689118106237254 2loss/SRqqt2/assignment_loss‚0.5770605048345749 2loss/FBt1/assignment_loss‚0.019155774132417713 1loss/FBt2/assignment_loss‚0.04983188400156928 1loss/SRqqt2/detection_loss‚0.9757211535362545 .loss/FBt1/detection_loss‚0.487848837225351 0loss/FRt1/assignment_loss‚0.8575579571612004 /loss/FRt1/detection_loss‚1.5682933755708692ý8P ‹… 2loss/SRqqt1/assignment_loss‚0.5689118106237254 2loss/SRqqt2/assignment_loss‚0.5770605048345749 2loss/FBt1/assignment_loss‚0.019155774132417713 1loss/FBt2/assignment_loss‚0.04983188400156928 1loss/SRqqt2/detection_loss‚0.9757211535362545 .loss/FBt1/detection_loss‚0.487848837225351 0loss/FRt1/assignment_loss‚0.8575579571612004 /loss/FRt1/detection_loss‚1.5682933755708692 trainer/global_step‚3999 0loss/FRt2/assignment_loss‚1.8197753395978902 /loss/FRt2/detection_loss‚0.5826529553830987 /loss/FBt2/detection_loss‚0.6750555617515656 0loss/SRqqt1/detection_loss‚0.682867293599167 %loss/total_loss‚8.864732447417683 epoch‚9 % _timestamp‚1.7843263758820047e+09 _runtime‚6629.256345238 _step‚328Èr·*%Œ‚(:’ èÑêÒ¬¯ gpu.0.gpu‚18.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.89299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pw(+3‚(:¨ èÑêÒáÏÅ°& disk.nvme0n1.out‚62508.86376953125# disk.dm-0.out‚62508.86376953125" proc.memory.availableMB‚1892.5! proc.memory.rssMB‚ 2890.484375 proc.cpu.threads‚17$ disk./.usageGB‚584.1313171386719" disk.nvme0n1.in‚13128.87890625# disk.rbd1.out‚17770.06787109375 disk.dm-0.in‚13128.87890625 disk.sda1.out‚0 network.recv‚ 1.692161e+06 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 416.61328125 disk.sda3.out‚ 416.61328125 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚13128.87890625 disk.sda.in‚ 0.8359375 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875( disk.nvme0c0n1.out‚62508.86376953125 disk.drbd2165.out‚0) proc.memory.percent‚17.64211654663086 cpu‚25.550695745857404) disk./.usagePercent‚39.20154281199659 disk.sda.out‚ 416.61328125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.8359375 disk.rbd2.in‚0.015625 disk.rbd2.out‚43.46240234375 disk.sda1.in‚0 disk.drbd2165.in‚0# memory_percent‚88.4490966796875 network.sent‚ 7.42637e+06ƒãŽ‚(:’ ÷ÑêÒ÷šë¬ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µôB¦6‚(:« ÷ÑêÒ”ÿ®$ disk./.usageGB‚584.1311378479004" disk.dm-0.out‚62518.6533203125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.8359375$ memory_percent‚88.48516941070557 network.sent‚ 7.489493e+06 disk.sda3.out‚ 417.08984375# disk.nvme0c0n1.in‚ 13217.7578125 disk.sda2.out‚0* proc.memory.percent‚17.678236961364746 cpu‚25.55696603521886 disk.rbd2.in‚0.015625 disk.sda1.out‚0# disk.rbd1.out‚17770.06787109375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.697313e+06) disk./.usagePercent‚39.20153077964248 disk.rbd2.out‚ 43.486328125' disk.nvme0c0n1.out‚62518.6533203125! disk.nvme0n1.in‚ 13217.7578125 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.8359375% disk.nvme0n1.out‚62518.6533203125 disk.sda.in‚ 0.8359375# proc.memory.rssMB‚ 2896.40234375 disk.dm-0.in‚ 13217.7578125 disk.dm-1.out‚ 417.08984375) proc.memory.availableMB‚ 1886.58984375 disk.rbd1.in‚ 506.4921875 disk.drbd2165.in‚0 disk.sda.out‚ 417.08984375 proc.cpu.threads‚17Û ¹Ò2- _wandbruntime‚6656  _runtime‚6656§™¼«è;‘Â{ @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14242497086524963 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6531398296356201 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7563895583152771 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15701411664485931 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939308166503906 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0939847007393837 'particle/accuracy‚0.844048261642456 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.818972110748291 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6068537831306458 'particle/f_score‚0.6359667778015137 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8046863079071045 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13610579073429108 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7563895583152771 *jet/accuracy_1_of_3‚0.8963028788566589 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6554813385009766 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8008124828338623 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7379941344261169 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5739486813545227 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.666102409362793 particle/accuracy_2_of_4‚1 particle/accuracy_2_of_5‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.931900680065155 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6415982246398926 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 _step‚329 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9923747777938843 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9865351915359497 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6374003291130066 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5765883326530457 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_5_of_5‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6494972705841064 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4685724377632141 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923602342605591 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6472845077514648 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005514815915375948 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6155183911323547 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15219596028327942 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8530320525169373 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.569897472858429 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5924723744392395 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27793702483177185 % _timestamp‚1.7843264028614988e+09 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6491393446922302 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14473339915275574 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4188876450061798 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0340517982840538 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5881425738334656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15058553218841553 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7413052916526794 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12647739052772522 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027084214612841606 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.129935622215271 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5739486813545227 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1316905915737152 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9014720916748047 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6623407006263733 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1470470279455185 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6990463733673096 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14477388560771942 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6433539390563965 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014584166929125786 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54666668176651 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14170002937316895 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023133274167776108 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8704730272293091 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6098456382751465 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5493288636207581 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934378862380981 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4533223509788513 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.2707696557044983 *jet/accuracy_1_of_4‚0.9599545001983643 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959147572517395 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6402104496955872 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3410899043083191 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.559906542301178 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1640922874212265 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950129985809326 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1313810497522354 particle/accuracy_3_of_4‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5779750943183899 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952874779701233 +particle/sensitivity‚0.6343739628791809 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2145860344171524 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20548759400844574 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.995344877243042 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 _runtime‚6656.237127473 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12615257501602173 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8047240972518921 particle/accuracy_1_of_3‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'loss/val_total_loss‚8.6097534074061 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3352077007293701 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8379629254341125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11317519098520279 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1313810497522354 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9843583703041077 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07853887975215912 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07291615754365921 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07837063074111938 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07829870283603668 *particle/specificity‚0.901387095451355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14788201451301575 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0786738470196724 particle/accuracy_2_of_2‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48996663093566895 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14599190652370453 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07291615754365921 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7320483922958374 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129626870155334 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.576732337474823 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6474232077598572 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.4862021803855896 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2560482621192932 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6896090507507324 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9844380617141724 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14583972096443176 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10037911683320999 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 *jet/accuracy_1_of_2‚0.7301610708236694 *jet/accuracy_2_of_4‚0.7234835624694824 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.614054262638092 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9840996265411377 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6208031177520752 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8103363513946533 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709635496139526 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12818686664104462 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6537478566169739 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13418059051036835 +jet/accuracy_3_of_3‚0.07020732015371323 /particle/accuracy_4_of_4‚0.9623150825500488 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.530656099319458 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.33175045251846313 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 trainer/global_step‚3999 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966698884963989 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6065492630004883 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00964359287172556 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2143009901046753 +jet/accuracy_2_of_2‚0.13297101855278015 particle/accuracy_1_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7004292011260986 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9955384135246277 8¦@´à Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19373080134391785 epoch‚9 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.661583662033081 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3385177254676819 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5641706585884094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6validation_average_jet_accuracy‚0.3193419873714447 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6545525789260864 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083134651184žÍƒÍ=’Ç{ =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14170002937316895 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023133274167776108 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8704730272293091 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6098456382751465 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5493288636207581 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934378862380981 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4533223509788513 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.2707696557044983 *jet/accuracy_1_of_4‚0.9599545001983643 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959147572517395 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6402104496955872 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3410899043083191 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.559906542301178 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1640922874212265 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950129985809326 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1313810497522354 particle/accuracy_3_of_4‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5779750943183899 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952874779701233 +particle/sensitivity‚0.6343739628791809 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2145860344171524 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20548759400844574 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.995344877243042 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 _runtime‚6656.237127473 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12615257501602173 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8047240972518921 particle/accuracy_1_of_3‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'loss/val_total_loss‚8.6097534074061 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3352077007293701 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8379629254341125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11317519098520279 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1313810497522354 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9843583703041077 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07853887975215912 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07291615754365921 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07837063074111938 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07829870283603668 *particle/specificity‚0.901387095451355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14788201451301575 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0786738470196724 particle/accuracy_2_of_2‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48996663093566895 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14599190652370453 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07291615754365921 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7320483922958374 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129626870155334 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.576732337474823 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6474232077598572 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.4862021803855896 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2560482621192932 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6896090507507324 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9844380617141724 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14583972096443176 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10037911683320999 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 *jet/accuracy_1_of_2‚0.7301610708236694 *jet/accuracy_2_of_4‚0.7234835624694824 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.614054262638092 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9840996265411377 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6208031177520752 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8103363513946533 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709635496139526 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12818686664104462 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6537478566169739 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13418059051036835 +jet/accuracy_3_of_3‚0.07020732015371323 /particle/accuracy_4_of_4‚0.9623150825500488 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.530656099319458 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.33175045251846313 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 trainer/global_step‚3999 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966698884963989 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6065492630004883 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00964359287172556 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2143009901046753 +jet/accuracy_2_of_2‚0.13297101855278015 particle/accuracy_1_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7004292011260986 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9955384135246277 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19373080134391785 epoch‚9 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.661583662033081 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3385177254676819 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5641706585884094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6validation_average_jet_accuracy‚0.3193419873714447 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6545525789260864 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083134651184 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14242497086524963 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6531398296356201 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7563895583152771 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15701411664485931 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939308166503906 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0939847007393837 'particle/accuracy‚0.844048261642456 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.818972110748291 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6068537831306458 'particle/f_score‚0.6359667778015137 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8046863079071045 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13610579073429108 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7563895583152771 *jet/accuracy_1_of_3‚0.8963028788566589 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6554813385009766 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8008124828338623 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7379941344261169 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5739486813545227 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.666102409362793 particle/accuracy_2_of_4‚1 particle/accuracy_2_of_5‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.931900680065155 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6415982246398926 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 _step‚329 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9923747777938843 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9865351915359497 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6374003291130066 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5765883326530457 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_5_of_5‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6494972705841064 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4685724377632141 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923602342605591 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6472845077514648 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005514815915375948 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6155183911323547 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15219596028327942 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8530320525169373 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.569897472858429 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5924723744392395 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27793702483177185 % _timestamp‚1.7843264028614988e+09 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6491393446922302 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14473339915275574 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4188876450061798 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0340517982840538 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5881425738334656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15058553218841553 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7413052916526794 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12647739052772522 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027084214612841606 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.129935622215271 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5739486813545227 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1316905915737152 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9014720916748047 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6623407006263733 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1470470279455185 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6990463733673096 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14477388560771942 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6433539390563965 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014584166929125786 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54666668176651 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129É|FÙ’“‚(:‡ †ÒêÒàǃ¬ gpu.0.gpu‚24.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3592 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sv–M6”‚(:« †ÒêÒ½‘ή& disk.nvme0n1.out‚62526.67431640625 disk.sda.in‚ 0.8359375 proc.cpu.threads‚17$ disk./.usageGB‚584.1348686218262$ disk.nvme0c0n1.in‚13286.19921875 disk.rbd2.out‚ 44.416015625 disk.dm-0.in‚13286.19921875# disk.dm-0.out‚62526.67431640625( proc.memory.availableMB‚ 2070.5703125# proc.memory.rssMB‚ 3704.96484375 network.recv‚ 1.701018e+06( disk.nvme0c0n1.out‚62526.67431640625 disk.sda1.in‚0 disk.dm-1.in‚ 0.8359375" disk.nvme0n1.in‚13286.19921875 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.dm-1.out‚ 417.3984375) disk./.usagePercent‚39.20178115501096 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 417.3984375# disk.rbd1.out‚17770.13037109375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.drbd2165.in‚0$ memory_percent‚87.36224174499512) proc.memory.percent‚22.61331081390381 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 417.3984375 cpu‚25.56117949905556 network.sent‚ 7.49481e+06S…‘y“•‚(:ˆ •ÒêÒ£·¦¬ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚251.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 pt=–‚(:² •ÒêÒ•ÞÀ¯# disk.rbd1.out‚17770.13037109375# disk.dm-0.out‚63502.14599609375 disk.drbd2165.out‚0 disk.sda3.in‚ 0.8359375$ disk.nvme0c0n1.in‚13286.19921875& disk.nvme0n1.out‚63502.14599609375 disk.rbd2.out‚ 44.5908203125 network.sent‚ 7.552704e+06 network.recv‚ 1.70598e+06* disk./.usagePercent‚39.201802147628776 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 417.69140625 disk.sda3.out‚ 417.69140625 disk.rbd2.in‚0.015625 disk.rbd1.in‚ 506.4921875 disk.sda1.out‚0( disk.nvme0c0n1.out‚63502.14599609375) proc.memory.availableMB‚ 2866.12109375# proc.memory.rssMB‚ 2909.63671875 disk.sda.out‚ 417.69140625" disk.nvme0n1.in‚13286.19921875 disk.sda.in‚ 0.8359375 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚82.50658512115479 disk.sda2.out‚0 disk.dm-0.in‚13286.19921875 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375* proc.memory.percent‚17.759013175964355 cpu‚25.559132444177145 disk.sda2.in‚0# disk./.usageGB‚584.135181427002{n!cŸ—‚(:” ¤ÒêÒõŽÀ« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.462' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.492399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘™1˜‚(:¦ ¤ÒêÒìᜮ disk.dm-0.in‚ 13287.03125# disk.dm-0.out‚64481.32373046875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 13287.03125 cpu‚25.559420067791965) disk./.usagePercent‚39.20154332401166# disk.rbd1.out‚17770.19287109375 disk.rbd2.out‚ 44.5908203125 network.sent‚ 7.556842e+06$ disk./.usageGB‚584.1313247680664 disk.rbd1.in‚ 506.4921875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚64481.32373046875 proc.cpu.threads‚17& disk.nvme0n1.out‚64481.32373046875$ memory_percent‚82.50212669372559 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 417.8671875( proc.memory.availableMB‚ 2866.8515625* proc.memory.percent‚17.759013175964355 disk.sda.out‚ 417.8671875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 417.8671875# proc.memory.rssMB‚ 2909.63671875 network.recv‚ 1.708767e+06 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 13287.03125 disk.sda.in‚ 0.8359375©5"••™‚(:Š ³ÒêÒÎ㪬 gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚210.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.0704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0* §Ô<š‚(:± ³ÒêÒ öµ¯ disk.drbd2165.out‚0 disk.sda3.out‚ 418.24609375$ memory_percent‚82.50370025634766$ disk.nvme0c0n1.in‚13287.98828125 disk.rbd2.in‚0.015625 disk.dm-0.in‚13287.98828125# disk.dm-0.out‚64493.49951171875 disk.dm-1.out‚ 418.24609375 proc.cpu.threads‚17$ disk./.usageGB‚584.1311073303223& disk.nvme0n1.out‚64493.49951171875 disk.dm-1.in‚ 0.8359375# disk.rbd1.out‚17770.19287109375 disk.rbd2.out‚44.64990234375& proc.memory.availableMB‚ 2866.59375* proc.memory.percent‚17.759013175964355 disk.sda2.out‚0 cpu‚25.559835309529593 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375 disk.sda.out‚ 418.24609375 disk.sda1.out‚0 network.sent‚ 7.562493e+06 network.recv‚ 1.712432e+06" disk.nvme0n1.in‚13287.98828125 disk.sda.in‚ 0.8359375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2909.63671875( disk./.usagePercent‚39.2015287315822( disk.nvme0c0n1.out‚64493.49951171875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 t Ož›‚(:“ ÂÒêÒçÍ« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ÏÔŸ:œ‚(:¯ ÂÒêÒÜùϯ disk.dm-1.out‚ 418.58984375 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 418.58984375& disk.nvme0n1.out‚64496.65966796875 disk.sda2.out‚0 network.recv‚ 1.715807e+06$ disk./.usageGB‚584.1307182312012 disk.rbd2.out‚46.13427734375 disk.rbd2.in‚0.015625" disk.nvme0c0n1.in‚ 13426.640625# disk.dm-0.out‚64496.65966796875 disk.dm-1.in‚ 0.8359375$ memory_percent‚82.50100612640381* proc.memory.percent‚17.759013175964355 proc.cpu.threads‚17 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17770.25537109375 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.560145409026127 disk.nvme0n1.in‚ 13426.640625 disk.sda1.out‚0 disk.dm-0.in‚ 13426.640625 disk.drbd2165.in‚0 disk.sda.in‚ 0.8359375 disk.sda2.in‚0) proc.memory.availableMB‚ 2867.03515625 disk.sda1.in‚0 network.sent‚ 7.567014e+06( disk.nvme0c0n1.out‚64496.65966796875 disk.sda.out‚ 418.58984375# proc.memory.rssMB‚ 2909.63671875) disk./.usagePercent‚39.20150261881371 disk.drbd2165.out‚0ÍPâ•‚(:Š ÑÒêÒ ò׬ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚336.753' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‚_O8ž‚(:­ ÑÒêÒà÷‡¯$ disk.nvme0c0n1.in‚13427.47265625# disk.rbd1.out‚17770.25537109375 disk.sda.in‚ 0.8359375 disk.drbd2165.out‚0 disk.dm-1.out‚ 419.26171875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.8359375$ memory_percent‚82.50248432159424# proc.memory.rssMB‚ 2909.63671875% disk.nvme0n1.out‚64504.4580078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375" disk.dm-0.out‚64504.4580078125 disk.sda.out‚ 419.26171875 disk.rbd0.in‚0 disk.rbd2.in‚0.015625) proc.memory.availableMB‚ 2866.79296875* proc.memory.percent‚17.759013175964355 disk.sda3.in‚ 0.8359375) disk./.usagePercent‚39.20151004303221" disk.nvme0n1.in‚13427.47265625 disk.rbd2.out‚ 46.3359375 disk.drbd2165.in‚0 cpu‚25.560534068275047 network.recv‚ 1.71918e+06 disk.sda1.out‚0' disk.nvme0c0n1.out‚64504.4580078125 proc.cpu.threads‚17$ disk./.usageGB‚584.1308288574219 disk.dm-0.in‚13427.47265625 network.sent‚ 7.572624e+06 disk.sda1.in‚0 disk.sda3.out‚ 419.261718759f™32Ÿ- _wandbruntime‚6739  _runtime‚6739w”‚É Ã ' lr-AdamW/pg1‚0.00025128843231420793 ' lr-AdamW/pg2‚0.00025128843231420793 trainer/global_step‚4049 % _timestamp‚1.7843264861002893e+09 _runtime‚6739.474902465 _step‚330_¨áÎ¡È ' lr-AdamW/pg1‚0.00025128843231420793 ' lr-AdamW/pg2‚0.00025128843231420793 trainer/global_step‚4049 % _timestamp‚1.7843264861002893e+09 _runtime‚6739.474902465 _step‚330Ê}ðd?2¢- _wandbruntime‚6739  _runtime‚6739+dEãõ£ùO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1164270273544192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3809311632896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35082015 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3802792255640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34558725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843264861014278e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319411452152832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21911968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚4049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34558725 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314493193023488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809311632896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168079559003648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.lËÇ arge_pool.allocated‚10392534 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630767 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1168080930566656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35072042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11519434 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4918659659776 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314492824932352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4918627220480 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1163766804770816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168079559003648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚04áí(¤þO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21911968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚4049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34558725 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314493193023488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809311632896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168079559003648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10392534 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630767 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1168080930566656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35072042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11519434 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4918659659776 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314492824932352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4918627220480 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1163766804770816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168079559003648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1164270273544192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3809311632896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35082015 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3802792255640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34558725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843264861014278e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319411452152832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657444 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11522062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1164548248100864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1164548274274304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314570679431168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843264875033445e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34566937 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647451 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34566931 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35090507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21917139 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4919833060352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1167849746871312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1167848376842910 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21916900 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168359856339456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35090507 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168358484769280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3810236668416 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1164548248100864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚4049 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1168359856339456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810236668416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34566937 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319490544923648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _runtime‚ 6740.87832371 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35080520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35080520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803715721886 _step‚332 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34566931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805060322320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10395072 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164044661121024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175£)Ñ•#§‡N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314570679431168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843264875033445e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34566937 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647451 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34566931 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35090507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21917139 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4919833060352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1167849746871312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1167848376842910 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21916900 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168359856339456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35090507 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168358484769280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3810236668416 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1164548248100864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚4049 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1168359856339456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810236668416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34566937 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319490544923648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _runtime‚ 6740.87832371 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35080520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35080520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803715721886 _step‚332 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34566931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805060322320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10395072 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164044661121024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657444 ICDeviceStats Ý·´xMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11522062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1164548248100864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1164548274274304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175ÌË€„ü2¨- _wandbruntime‚6740  _runtime‚6740ƒ\]ü ©ƒ 0loss/FRt1/assignment_loss‚0.7461898469589126 0loss/FRt2/assignment_loss‚1.7743056565376627 2loss/FBt2/assignment_loss‚0.018965959833095746 1loss/SRqqt1/detection_loss‚0.6382423082485345 1loss/SRqqt2/detection_loss‚0.8569117914628366 /loss/FRt2/detection_loss‚0.6070481301722671 /loss/FBt1/detection_loss‚0.3294147957918526 1loss/FBt1/assignment_loss‚0.04861288205640262 %loss/total_loss‚8.197918269889346 epoch‚10 trainer/global_step‚4049 % _timestamp‚1.7843264875836093e+09 _runtime‚6740.958246673 _step‚333 2loss/SRqqt1/assignment_loss‚0.5336608862440723 2loss/SRqqt2/assignment_loss‚0.6365228625549709 /loss/FRt1/detection_loss‚1.4573759174197387 /loss/FBt2/detection_loss‚0.5506672326090007© §ðªˆ %loss/total_loss‚8.197918269889346 epoch‚10 trainer/global_step‚4049 % _timestamp‚1.7843264875836093e+09 _runtime‚6740.958246673 _step‚333 2loss/SRqqt1/assignment_loss‚0.5336608862440723 2loss/SRqqt2/assignment_loss‚0.6365228625549709 /loss/FRt1/detection_loss‚1.4573759174197387 /loss/FBt2/detection_loss‚0.5506672326090007 0loss/FRt1/assignment_loss‚0.7461898469589126 0loss/FRt2/assignment_loss‚1.7743056565376627 2loss/FBt2/assignment_loss‚0.018965959833095746 1loss/SRqqt1/detection_loss‚0.6382423082485345 1loss/SRqqt2/detection_loss‚0.8569117914628366 /loss/FRt2/detection_loss‚0.6070481301722671 /loss/FBt1/detection_loss‚0.3294147957918526 1loss/FBt1/assignment_loss‚0.04861288205640262ͽ£ü”«‚(:‰ àÒêÒò¬ç« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð~ê=3¬‚(:¨ àÒêÒ¢ó˜¯ network.sent‚ 7.577154e+06 disk.sda3.out‚ 419.578125 disk.sda1.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.759013175964355 network.recv‚ 1.722605e+06# disk.dm-0.out‚64514.85400390625 disk.dm-1.out‚ 419.578125* disk./.usagePercent‚39.201517979265766! disk.nvme0n1.in‚ 13428.4296875 disk.sda.in‚ 0.8359375 disk.sda.out‚ 419.578125 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.8359375 proc.cpu.threads‚17 disk.sda3.in‚ 0.8359375 disk.sda2.in‚0# disk.rbd1.out‚17770.33349609375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 13428.4296875 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.rbd2.out‚ 46.3359375 disk.dm-0.in‚ 13428.4296875# proc.memory.rssMB‚ 2909.63671875( disk.nvme0c0n1.out‚64514.85400390625& disk.nvme0n1.out‚64514.85400390625 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚82.50269889831543( proc.memory.availableMB‚ 2866.7578125 cpu‚25.560804919724706$ disk./.usageGB‚584.1309471130371 disk.drbd2165.out‚0€¥ñ”­‚(:‰ ïÒê҈̶® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#uùí(®‚(: ïÒêÒ±Úô® disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 13428.4296875# disk.nvme0n1.out‚64518.49609375 disk.sda.in‚ 0.8359375" memory_percent‚82.501220703125 proc.memory.availableMB‚2867 disk.sda1.out‚0 disk.sda3.out‚ 420.19140625 disk.sda2.out‚0* proc.memory.percent‚17.759013175964355 proc.cpu.threads‚17 network.sent‚ 7.640839e+06 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.rbd2.out‚46.40771484375 disk.dm-0.out‚64518.49609375# proc.memory.rssMB‚ 2909.63671875$ disk./.usageGB‚584.1310272216797 disk.dm-1.out‚ 420.19140625 disk.dm-0.in‚ 13428.4296875 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375% disk.nvme0c0n1.out‚64518.49609375 network.recv‚ 1.727877e+06) disk./.usagePercent‚39.20152335542399 disk.sda3.in‚ 0.8359375# disk.rbd1.out‚17770.33349609375 disk.rbd2.in‚0.015625# disk.nvme0c0n1.in‚ 13428.4296875 disk.sda.out‚ 420.19140625 disk.sda2.in‚0 disk.rbd0.in‚0 cpu‚25.56126201740491GÌ‚(:‰ þÒêÒÖÛ¹« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ÈU9°‚(:® þÒêÒôˆË®* proc.memory.percent‚17.759013175964355 disk.dm-0.in‚13631.72265625 disk.drbd2165.out‚0# disk.rbd1.out‚17770.39599609375 disk.rbd0.out‚0 disk.rbd2.out‚46.40771484375! disk.dm-0.out‚64529.166015625 network.sent‚ 7.645464e+06* disk./.usagePercent‚39.201522331393846$ disk./.usageGB‚584.1310119628906$ memory_percent‚82.50124454498291 disk.sda.in‚ 0.8359375 disk.sda.out‚ 420.43359375 disk.sda2.out‚0 disk.rbd2.in‚0.015625) proc.memory.availableMB‚ 2866.99609375 proc.cpu.threads‚17 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 420.43359375$ disk.nvme0c0n1.in‚13631.72265625 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 420.43359375 cpu‚25.56157037314182& disk.nvme0c0n1.out‚64529.166015625# proc.memory.rssMB‚ 2909.63671875 disk.drbd2165.in‚0 network.recv‚ 1.73133e+06" disk.nvme0n1.in‚13631.72265625$ disk.nvme0n1.out‚64529.166015625 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.sda1.in‚0SàÝ\Ÿ±‚(:” ÓêÒøÉÝ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨uåj/²‚(:¤ ÓêÒÄÏé®$ disk.nvme0c0n1.in‚13631.72265625$ disk.nvme0n1.out‚64533.560546875* proc.memory.percent‚17.759013175964355 disk.sda.in‚ 0.8359375 disk.sda2.in‚0! disk.dm-0.out‚64533.560546875 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 420.78125& disk.nvme0c0n1.out‚64533.560546875 network.sent‚ 7.65104e+06 disk.sda3.in‚ 0.8359375 disk.dm-1.out‚ 420.78125 network.recv‚ 1.734651e+06 disk.sda1.in‚0$ memory_percent‚82.50272274017334 cpu‚25.56209286312721$ disk./.usageGB‚584.1314582824707 disk.rbd0.out‚0) proc.memory.availableMB‚ 2866.75390625" disk.nvme0n1.in‚13631.72265625 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17770.39599609375 disk.rbd0.in‚0 disk.rbd2.out‚46.47021484375 disk.dm-0.in‚13631.72265625 disk.drbd2165.in‚0) disk./.usagePercent‚39.20155228427535 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda.out‚ 420.78125 disk.sda2.out‚0# proc.memory.rssMB‚ 2909.63671875 proc.cpu.threads‚17!Ê–3’³‚(:‡ œÓêÒñ¼Õ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àŠ9´‚(:® œÓêÒã½ù­$ disk./.usageGB‚584.1314811706543 disk.sda.in‚ 0.8359375& disk.nvme0n1.out‚64546.24365234375 cpu‚25.562365888652092 disk.rbd2.out‚46.47021484375 disk.dm-0.in‚ 13632.6796875 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375( disk.nvme0c0n1.out‚64546.24365234375( proc.memory.availableMB‚ 2866.7890625# proc.memory.rssMB‚ 2909.63671875 disk.sda1.out‚0 disk.dm-1.out‚ 421.0078125 disk.sda.out‚ 421.0078125 disk.sda2.in‚0# disk.rbd1.out‚17770.45849609375$ memory_percent‚82.50250816345215 network.sent‚ 7.656253e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚ 1.738175e+06) disk./.usagePercent‚39.20155382032056 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 13632.6796875 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0# disk.dm-0.out‚64546.24365234375* proc.memory.percent‚17.759013175964355 disk.sda3.out‚ 421.0078125! disk.nvme0n1.in‚ 13632.6796875©Úå®2µ- _wandbruntime‚6818  _runtime‚6818õÉ6ɶà _runtime‚6818.640623529 _step‚334 ' lr-AdamW/pg1‚0.00025067844159364186 ' lr-AdamW/pg2‚0.00025067844159364186 trainer/global_step‚4099 % _timestamp‚1.7843265652660072e+09ìNýÑÎ·È ' lr-AdamW/pg2‚0.00025067844159364186 trainer/global_step‚4099 % _timestamp‚1.7843265652660072e+09 _runtime‚6818.640623529 _step‚334 ' lr-AdamW/pg1‚0.00025067844159364186Î+.©Ð2¸- _wandbruntime‚6818  _runtime‚6818¤%ÞÖÿ'¹ùO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3856972953088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1177660893679616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1178170283876352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843265652671514e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35497794 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34969031 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3849029368748 _runtime‚6818.641927601 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10519679 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1181509897620396 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3855627563520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35507767 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚335 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1182027283002880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1178170283876352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318367513407488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3850373969070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3855627563520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11656748 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182025911439872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1178170310049792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11656998 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318367145316352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476798 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34969025 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Õ»]²(ºþO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182025911439872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1178170310049792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11656998 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318367145316352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476798 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34969025 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3856972953088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1177660893679616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1178170283876352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843265652671514e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35497794 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34969031 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3849029368748 _runtime‚6818.641927601 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10519679 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1181509897620396 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3855627563520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35507767 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚335 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1182027283002880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1178170283876352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318367513407488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3850373969070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3855627563520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11656748 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51ÏÿÖY2»- _wandbruntime‚6820  _runtime‚6820ÈM÷Œ¼ƒN NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35516259 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1178448284606464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35516259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318444999815168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483503 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35506272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4099 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1181788702864818 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22181918 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843265666534975e+09 Q} Monitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11659448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323424419244032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22181670 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1178448310779904 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10522227 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1181790072893220 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4979822049280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182306208775680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11659691 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3857897995776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4979787520000 _runtime‚6820.028353978 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1177938775457792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178448284606464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34977231 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3856552599040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182306208775680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1178448310779904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856552599040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35506272 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3849952834994 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3851297435428 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640Oå£Ð'½ˆN QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22181670 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1178448310779904 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10522227 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1181790072893220 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4979822049280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182306208775680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11659691 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3857897995776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4979787520000 _runtime‚6820.028353978 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1177938775457792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178448284606464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34977231 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3856552599040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182306208775680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1178448310779904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856552599040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35506272 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3849952834994 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3851297435428 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35516259 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1178448284606464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35516259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318444999815168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483503 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35506272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4099 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1181788702864818 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22181918 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843265666534975e+09ÐcO¨†2¾- _wandbruntime‚6820  _runtime‚6820NAu®¿ý $ _timestamp‚1.784326566739869e+09 0loss/FRt2/assignment_loss‚1.8203818970651768 2loss/SRqqt1/assignment_loss‚0.5145301819720225 _runtime‚ 6820.11422553 _step‚337 epoch‚10 1loss/FBt1/assignment_loss‚0.01601884743536118 /loss/FRt1/detection_loss‚1.6429451722730266 /loss/SRqqt1/detection_loss‚0.66120395207742 1loss/SRqqt2/detection_loss‚0.9397100632275177 1loss/FBt2/assignment_loss‚0.07574765686992653 /loss/FBt2/detection_loss‚0.7598006656238189 %loss/total_loss‚8.978717848988204 0loss/FRt1/assignment_loss‚0.7372977008302528 1loss/SRqqt2/assignment_loss‚0.575667885860849 /loss/FRt2/detection_loss‚0.5678565832606173 /loss/FBt1/detection_loss‚0.6675572424922138 trainer/global_step‚4099p–RÀ‚ 0loss/FRt2/assignment_loss‚1.8203818970651768 2loss/SRqqt1/assignment_loss‚0.5145301819720225 _runtime‚ 6820.11422553 _step‚337 epoch‚10 1loss/FBt1/assignment_loss‚0.01601884743536118 /loss/FRt1/detection_loss‚1.6429451722730266 /loss/SRqqt1/detection_loss‚0.66120395207742 1loss/SRqqt2/detection_loss‚0.9397100632275177 1loss/FBt2/assignment_loss‚0.07574765686992653 /loss/FBt2/detection_loss‚0.7598006656238189 %loss/total_loss‚8.978717848988204 0loss/FRt1/assignment_loss‚0.7372977008302528 1loss/SRqqt2/assignment_loss‚0.575667885860849 /loss/FRt2/detection_loss‚0.5678565832606173 /loss/FBt1/detection_loss‚0.6675572424922138 trainer/global_step‚4099 $ _timestamp‚1.784326566739869e+09Ñ®”vâ”Á‚(:‰ «ÓêÒ÷ÍѬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û7Lx5‚(:ª «ÓêÒ×ϯ disk.sda.in‚ 0.8359375 disk.sda3.in‚ 0.8359375% disk.nvme0n1.out‚64555.7666015625 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17770.45849609375 disk.rbd0.in‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 421.359375 network.sent‚ 7.661305e+06 network.recv‚ 1.741806e+06* disk./.usagePercent‚39.201551004237686 disk.rbd2.out‚ 46.494140625 disk.sda1.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda.out‚ 421.359375 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚82.50393867492676 disk.sda2.in‚0* proc.memory.percent‚17.759013175964355 cpu‚25.562654201657114 disk.dm-0.in‚13633.51171875$ disk./.usageGB‚584.1314392089844 disk.sda3.out‚ 421.359375' disk.nvme0c0n1.out‚64555.7666015625" disk.nvme0n1.in‚13633.51171875 disk.sda2.out‚0( proc.memory.availableMB‚ 2866.5546875# proc.memory.rssMB‚ 2909.63671875 proc.cpu.threads‚17 disk.drbd2165.in‚0" disk.dm-0.out‚64555.7666015625$ disk.nvme0c0n1.in‚13633.51171875Û¬£ŸÃ‚(:” ºÓêÒ—Õ©¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚250.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ÃaÜ4Ä‚(:© ºÓêÒÙ°$ disk./.usageGB‚584.1317329406738" disk.nvme0c0n1.in‚ 13751.890625 disk.rbd0.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2867.01171875 network.sent‚ 7.723916e+06 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 421.62109375' disk.nvme0c0n1.out‚64561.4560546875 disk.rbd0.out‚0 network.recv‚ 1.746929e+06% disk.nvme0n1.out‚64561.4560546875 disk.sda2.in‚0 disk.dm-0.in‚ 13751.890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 421.62109375 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.8359375# disk.rbd1.out‚17770.52099609375) disk./.usagePercent‚39.20157071681782 disk.nvme0n1.in‚ 13751.890625" disk.dm-0.out‚64561.4560546875 disk.sda.in‚ 0.8359375 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.dm-1.out‚ 421.62109375$ memory_percent‚82.50114917755127# proc.memory.rssMB‚ 2909.63671875 cpu‚25.56302406642641 proc.cpu.threads‚17 disk.rbd2.out‚ 47.240234375 disk.drbd2165.out‚0* proc.memory.percent‚17.759013175964355kvî;Å‚(:° ÉÓêҮǯ# proc.memory.rssMB‚ 2909.63671875) disk./.usagePercent‚39.20156150054659 disk.sda.out‚ 421.98828125 disk.rbd2.out‚ 47.3505859375 disk.dm-1.out‚ 421.98828125 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚82.50155448913574 proc.cpu.threads‚17$ disk./.usageGB‚584.1315956115723$ disk.nvme0c0n1.in‚13752.84765625 disk.sda.in‚ 0.8359375 disk.dm-0.in‚13752.84765625* proc.memory.percent‚17.759013175964355 network.recv‚ 1.750636e+06' disk.nvme0c0n1.out‚64569.0341796875# disk.rbd1.out‚17770.52099609375 disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.8359375 disk.drbd2165.out‚0 cpu‚25.563399156065874 disk.drbd2165.in‚0 network.sent‚ 7.728973e+06 disk.sda3.out‚ 421.98828125% disk.nvme0n1.out‚64569.0341796875 disk.sda2.in‚0( proc.memory.availableMB‚ 2866.9453125 disk.dm-1.in‚ 0.8359375" disk.nvme0n1.in‚13752.84765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0" disk.dm-0.out‚64569.0341796875Ër0v”Æ‚(:‰ ÉÓêÒ迾® gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ocÇ‚(:’ ØÓêÒ¬ªÕ¬ gpu.0.gpu‚20.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚321.249' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.24980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÉCQ5È‚(:ª ØÓêÒº€°®! disk.dm-0.out‚64578.748046875 disk.dm-1.out‚ 422.28515625* proc.memory.percent‚17.759013175964355 cpu‚25.56377781572538 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 48.5498046875 disk.drbd2165.out‚0 disk.sda3.out‚ 422.28515625& disk.nvme0c0n1.out‚64578.748046875# proc.memory.rssMB‚ 2909.63671875 proc.cpu.threads‚17 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 13753.6796875 disk.rbd0.out‚0 network.recv‚ 1.754037e+06$ disk./.usageGB‚584.1317443847656 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17770.58349609375! disk.nvme0n1.in‚ 13753.6796875$ disk.nvme0n1.out‚64578.748046875 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda.in‚ 0.8359375 disk.dm-0.in‚ 13753.6796875 disk.sda1.out‚0$ memory_percent‚82.50157833099365) proc.memory.availableMB‚ 2866.94140625 disk.sda.out‚ 422.28515625 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.sda3.in‚ 0.8359375 network.sent‚ 7.734381e+06) disk./.usagePercent‚39.20157148484042ïöý>ŸÉ‚(:” çÓêÒꤑ® gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀI#È0Ê‚(:¥ çÓêÒñÃí®! disk.nvme0n1.in‚ 13753.6796875$ disk.nvme0n1.out‚64581.837890625! disk.dm-0.out‚64581.837890625# disk.nvme0c0n1.in‚ 13753.6796875& disk.nvme0c0n1.out‚64581.837890625 disk.sda2.out‚0 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 422.7265625 disk.rbd2.in‚0.015625 disk.sda1.out‚0& proc.memory.availableMB‚ 2867.78125# proc.memory.rssMB‚ 2909.63671875 network.recv‚ 1.75756e+06 disk.sda.out‚ 422.7265625 disk.rbd0.in‚0 disk.rbd2.out‚48.66845703125 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚82.49645233154297 disk.dm-1.in‚ 0.8359375# disk.rbd1.out‚17770.58349609375 disk.rbd0.out‚0 disk.dm-0.in‚ 13753.6796875 disk.sda.in‚ 0.8359375 disk.sda2.in‚0 cpu‚25.56406979288209 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.out‚ 422.7265625$ disk./.usageGB‚584.1312103271484 disk.rbd1.in‚ 506.52734375* proc.memory.percent‚17.759013175964355 network.sent‚ 7.739443e+06* disk./.usagePercent‚39.201535643785625Ȧ2Ë- _wandbruntime‚6899  _runtime‚68996îCÇÌÁ & lr-AdamW/pg1‚0.0002500611516518813 & lr-AdamW/pg2‚0.0002500611516518813 trainer/global_step‚4149 % _timestamp‚1.7843266458291602e+09 _runtime‚6899.203787222 _step‚338ˆÈL–ÌÍÆ _runtime‚6899.203787222 _step‚338 & lr-AdamW/pg1‚0.0002500611516518813 & lr-AdamW/pg2‚0.0002500611516518813 trainer/global_step‚4149 % _timestamp‚1.7843266458291602e+09ÒœQ Ç2Î- _wandbruntime‚6899  _runtime‚6899£ÎW ©ÏùO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35379331 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192070320381952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195972263876096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3896611082178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322241465700352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1195450223642304 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192070346555392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3901943494144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_××jOVpool.freed‚10646829 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322241833791488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35933519 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35923546 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195973635439104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901943494144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3903288883712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195972263876096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11793171 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843266458303788e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195973635439104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚6899.205159904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0W´ƒR(ÐþO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195972263876096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3896611082178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322241465700352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1195450223642304 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192070346555392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3901943494144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10646829 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322241833791488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35933519 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35923546 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195973635439104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901943494144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3903288883712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195972263876096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11793171 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843266458303788e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195973635439104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚6899.205159904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35379331 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192070320381952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175Ó|ƒ’Ñ‚(:‡ öÓêҟ〬 gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>£¢9Ò‚(:® öÓêÒß…‰¯ disk.sda2.in‚0# disk.dm-0.out‚64591.48193359375 disk.drbd2165.out‚0" disk.nvme0n1.in‚14166.26171875 disk.dm-0.in‚14166.26171875 disk.sda3.in‚ 0.8359375# proc.memory.rssMB‚ 2909.63671875 proc.cpu.threads‚17 disk.sda.in‚ 0.8359375 disk.rbd1.in‚ 506.52734375 disk.sda1.out‚0 disk.sda3.out‚ 422.94921875 cpu‚25.5642181327292) disk./.usagePercent‚39.20160374178974 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 422.94921875$ disk.nvme0c0n1.in‚14166.26171875$ memory_percent‚88.42964172363281% proc.memory.availableMB‚ 1895.6875 network.recv‚ 1.761004e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375$ disk./.usageGB‚584.1322250366211& disk.nvme0n1.out‚64591.48193359375 disk.sda.out‚ 422.94921875# disk.rbd1.out‚17770.64599609375 disk.rbd0.out‚0 disk.rbd2.out‚48.66845703125 network.sent‚ 7.74455e+06 disk.sda2.out‚0( disk.nvme0c0n1.out‚64591.48193359375* proc.memory.percent‚17.759013175964355ežÝ2Ó- _wandbruntime‚6900  _runtime‚6900p 'ÔƒN LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1191832838938624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329548 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚4149 _step‚340 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35942011 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3902868529664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35387537 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322318952108032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3897534548536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195729028886726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35387531 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1191832864366592 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329548 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _runtime‚6900.605299483 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22445279 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11796176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35387537 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1196252561211904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1192348347285504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22445553 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35932024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10649377 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327358500703232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843266472303646e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195730398915128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322319320199168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3904213926400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1196251189641728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5039141780480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1192348321112064 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1196252561211904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327358093888512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5039180504064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319555 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3904213926400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192348347285504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0øŒ±´oÕˆN YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3904213926400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192348347285504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1191832838938624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329548 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚4149 _step‚340 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35942011 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3902868529664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35387537 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322318952108032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3897534548536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195729028886726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35387531 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1191832864366592 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329548 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _runtime‚6900.605299483 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22445279 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11796176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.l©²ª:Ÿarge_pool.allocated‚35387537 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1196252561211904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1192348347285504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22445553 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35932024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10649377 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327358500703232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843266472303646e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195730398915128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322319320199168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3904213926400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1196251189641728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5039141780480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1192348321112064 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1196252561211904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327358093888512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5039180504064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319555 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995ÔùŠ52Ö- _wandbruntime‚6900  _runtime‚6900v"ïƒ ×ƒ trainer/global_step‚4149 % _timestamp‚1.7843266473107493e+09 0loss/FRt1/assignment_loss‚0.7068307211450047 0loss/FRt2/assignment_loss‚1.7645351104090148 2loss/SRqqt2/assignment_loss‚0.5838368736819605 1loss/SRqqt2/detection_loss‚0.7755299957619319 %loss/total_loss‚8.373396079699093 epoch‚10 _runtime‚6900.685358557 _step‚341 /loss/FRt1/detection_loss‚1.5177918891057443 /loss/FBt2/detection_loss‚0.6969897577610726 2loss/SRqqt1/assignment_loss‚0.5372127528746927 2loss/FBt1/assignment_loss‚0.017617113444956645 1loss/FBt2/assignment_loss‚0.04761345824121795 /loss/FRt2/detection_loss‚0.5447834383993136 1loss/SRqqt1/detection_loss‚0.6225650497467914 /loss/FBt1/detection_loss‚0.5580899191273918þð´f؈ 2loss/SRqqt1/assignment_loss‚0.5372127528746927 2loss/FBt1/assignment_loss‚0.017617113444956645 1loss/FBt2/assignment_loss‚0.04761345824121795 /loss/FRt2/detection_loss‚0.5447834383993136 1loss/SRqqt1/detection_loss‚0.6225650497467914 /loss/FBt1/detection_loss‚0.5580899191273918 trainer/global_step‚4149 % _timestamp‚1.7843266473107493e+09 0loss/FRt1/assignment_loss‚0.7068307211450047 0loss/FRt2/assignment_loss‚1.7645351104090148 2loss/SRqqt2/assignment_loss‚0.5838368736819605 1loss/SRqqt2/detection_loss‚0.7755299957619319 %loss/total_loss‚8.373396079699093 epoch‚10 _runtime‚6900.685358557 _step‚341 /loss/FRt1/detection_loss‚1.5177918891057443 /loss/FBt2/detection_loss‚0.6969897577610726Õ«CÒ¾”Ù‚(:‰ …ÔêÒÝ懬 gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡8Ú=Ú‚(:² …ÔêÒðÖÒ® disk.sda2.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚64599.46435546875) proc.memory.availableMB‚ 1895.69140625 network.sent‚ 7.797446e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.20160886194042 disk.sda.in‚ 0.8359375# disk.rbd1.out‚17770.64599609375 disk.sda1.out‚0# memory_percent‚88.4296178817749 disk.sda3.in‚ 0.8359375 disk.sda.out‚ 423.25390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 423.25390625 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625& disk.nvme0n1.out‚64599.46435546875# proc.memory.rssMB‚ 2909.63671875 disk.sda3.out‚ 423.25390625" disk.nvme0n1.in‚14219.86328125* proc.memory.percent‚17.759013175964355 cpu‚25.564478876067376$ disk.nvme0c0n1.in‚14219.86328125$ disk./.usageGB‚584.1323013305664 disk.rbd2.out‚ 48.759765625 disk.dm-0.in‚14219.86328125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚64599.46435546875 network.recv‚ 1.766183e+06s¦1œÛ‚(:… ”ÔêÒŒÄȬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öh“¤1Ü‚(:¦ ”ÔêÒýÜí¯$ memory_percent‚88.42887878417969* proc.memory.percent‚17.759013175964355 cpu‚25.56486988648781 disk.sda.out‚ 423.796875# disk.rbd1.out‚17770.77099609375 disk.rbd2.in‚0.015625& disk.nvme0n1.out‚64604.97412109375 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 423.796875% proc.memory.availableMB‚ 1895.8125" disk.nvme0n1.in‚14219.86328125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚14219.86328125$ disk.nvme0c0n1.in‚14219.86328125# disk./.usageGB‚584.132209777832 disk.sda.in‚ 0.8359375 disk.sda1.out‚0 disk.sda3.out‚ 423.796875 network.recv‚ 1.770968e+06# disk.dm-0.out‚64604.97412109375 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375 disk.sda2.out‚0 disk.rbd2.out‚ 48.759765625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚64604.97412109375# proc.memory.rssMB‚ 2909.63671875 proc.cpu.threads‚17 network.sent‚ 7.849448e+06( disk./.usagePercent‚39.2016027177596 disk.rbd1.in‚ 506.52734375 disk.drbd2165.out‚0Q‰ ³”Ý‚(:‰ £ÔêÒ…œ´« gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚186.939' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Æñ£*Þ‚(:Ÿ £ÔêÒ¾ŽÏ¯ disk.rbd2.out‚48.82275390625 cpu‚25.565090728188082 network.recv‚ 1.774711e+06* disk./.usagePercent‚39.201596573578776$ disk./.usageGB‚584.1321182250977% disk.nvme0n1.out‚64612.6181640625 disk.sda2.in‚0 disk.dm-1.out‚ 424.09375 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17770.77099609375 network.sent‚ 7.854513e+06 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 424.09375* proc.memory.percent‚17.759013175964355 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 14220.8203125 disk.sda.in‚ 0.8359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0# memory_percent‚88.4307861328125# disk.nvme0c0n1.in‚ 14220.8203125# proc.memory.rssMB‚ 2909.63671875 disk.sda.out‚ 424.09375" disk.dm-0.out‚64612.6181640625 disk.sda1.out‚0 disk.dm-1.in‚ 0.8359375' disk.nvme0c0n1.out‚64612.6181640625! disk.nvme0n1.in‚ 14220.8203125" proc.memory.availableMB‚1895.5%«í žß‚(:“ ²ÔêÒ¤ô¬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.579' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.31580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™††•0à‚(:¥ ²ÔêÒ£¾Ò¯ disk.sda.in‚ 0.8359375# disk.rbd1.out‚17770.83349609375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 424.26171875# memory_percent‚88.4307861328125 disk.rbd1.in‚ 506.52734375 disk.sda1.out‚0% disk.nvme0n1.out‚64618.9345703125# proc.memory.rssMB‚ 2909.63671875) disk./.usagePercent‚39.20159017339042 disk.sda2.in‚0 disk.dm-0.in‚ 14220.8203125 network.recv‚ 1.777973e+06 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375* proc.memory.percent‚17.759013175964355 disk.dm-1.out‚ 424.26171875" proc.memory.availableMB‚1895.5# disk./.usageGB‚584.132022857666 disk.sda.out‚ 424.26171875! disk.nvme0n1.in‚ 14220.8203125 network.sent‚ 7.860136e+06 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 14220.8203125' disk.nvme0c0n1.out‚64618.9345703125 disk.rbd0.out‚0 disk.rbd2.out‚48.82275390625" disk.dm-0.out‚64618.9345703125 disk.drbd2165.in‚0 cpu‚25.56527310967115 proc.cpu.threads‚17œ.ºá‚(:… ÁÔêÒñ‰Ø« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÂ.ò:â‚(:¯ ÁÔêÒ¾Û®* proc.memory.percent‚17.759013175964355 network.sent‚ 7.865297e+06# disk./.usageGB‚584.132080078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.rbd1.out‚17770.83349609375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625) disk./.usagePercent‚39.20159401350343# disk.dm-0.out‚64626.21923828125 disk.sda1.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda2.out‚0 disk.dm-0.in‚14395.16015625 disk.sda3.out‚ 424.6953125 proc.cpu.threads‚17 network.recv‚ 1.781262e+06$ disk.nvme0c0n1.in‚14395.16015625( disk.nvme0c0n1.out‚64626.21923828125& disk.nvme0n1.out‚64626.21923828125 disk.rbd0.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2909.63671875 cpu‚25.56574723519525 disk.sda1.out‚0 disk.sda.in‚ 0.8359375 disk.sda.out‚ 424.6953125 disk.rbd2.out‚ 48.8857421875 disk.dm-1.out‚ 424.6953125 disk.sda3.in‚ 0.8359375" disk.nvme0n1.in‚14395.16015625) proc.memory.availableMB‚ 1895.49609375 disk.rbd1.in‚ 506.52734375$ memory_percent‚88.43080997467041l4é2ã- _wandbruntime‚6979  _runtime‚6979rlÇäÁ % _timestamp‚1.7843267263849928e+09 _runtime‚6979.759595519 _step‚342 & lr-AdamW/pg1‚0.0002494366021363058 & lr-AdamW/pg2‚0.0002494366021363058 trainer/global_step‚4199ÒÓº”ÌåÆ & lr-AdamW/pg2‚0.0002494366021363058 trainer/global_step‚4199 % _timestamp‚1.7843267263849928e+09 _runtime‚6979.759595519 _step‚342 & lr-AdamW/pg1‚0.0002494366021363058Öð­B×2æ- _wandbruntime‚6979  _runtime‚6979!= Æÿ'çùO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35789625 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36349298 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3949604814336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3948259424768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36359271 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36349298 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326115786084352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3942848195286 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1205970356887552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35789625 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11929638 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326116154175488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6979.760862638 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5096863094784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1205449046069248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1205970356887552 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1205970383060992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22703622 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11929362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5096901825536 trainer/global_step‚4199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22703341 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72148902 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331213056001024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10773984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331212649179136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3949604814336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36359271 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1209918616312320 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1205970383060992 % _timestamp‚1.7843267263860636e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35789631 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1209390549664212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1205449071497216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 Ix® èþO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22703341 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72148902 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331213056001024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10773984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331212649179136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3949604814336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36359271 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1209918616312320 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1205970383060992 % _timestamp‚1.7843267263860636e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35789631 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1209390549664212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1205449071497216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35789625 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11929638 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326116154175488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6979.760862638 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5096863094784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1205449046069248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1205970356887552 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1205970383060992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22703622 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11929362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5096901825536 trainer/global_step‚4199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0×tÌB42é- _wandbruntime‚6981  _runtime‚6981T? 'êƒN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35797837 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36357776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1206248383791104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35797831 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36367763 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1205726953275392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843267277831755e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 trainer/global_step‚4199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1210198913648128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22708870 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210198913648128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331291309535232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209670724937036 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5098075766784 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22708597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5098037043200 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331291716349952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35797831 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3950529857024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3950529857024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3942427061210 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1205726927847424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10776522 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326193272492032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36367763 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165600 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35797837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚6981.158300734 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1206248383791104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3949184460288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11932343 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1206248383791104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35797831 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36367763 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1205726953275392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843267277831755e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 trainer/global_step‚4199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1210198913648128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22708870 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210198913648128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331291309535232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209670724937036 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5098075766784 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22708597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5098037043200 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331291716349952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35797831 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3950529857024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3950529857024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3942427061210 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1205726927847424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10776522 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326193272492032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36367763 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165600 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35797837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚6981.158300734 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1206248383791104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3949184460288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11932343 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223864968748544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3994575355392 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1219870419566592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1223866340311552 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1219870419566592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36199925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10901134 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329990106468352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12065775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335146216718336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36785023 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335145809896448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1223330875686120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36775050 trainer/global_step‚4249 % _timestamp‚1.7843268061427896e+09 _runtime‚7059.517604275 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12066055 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1219343160406016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10901129 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚347 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3995920744960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1223332245714410 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22966904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0é.ùh(þþO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72984954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223864968748544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3994575355392 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1219870419566592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1223866340311552 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1219870419566592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36199925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10901134 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329990106468352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12065775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335146216718336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36785023 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335145809896448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1223330875686120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36775050 trainer/global_step‚4249 % _timestamp‚1.7843268061427896e+09 _runtime‚7059.517604275 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12066055 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1219343160406016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10901129 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚347 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3995920744960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1223332245714410 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22966904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175Ûñ –ä2ÿ- _wandbruntime‚7060  _runtime‚7060®î'€‚N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330067592876032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1224145266084352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1223609680930542 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1220148394123264 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1220148394123264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12068507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1219621016756224 $ _timestamp‚1.784326807526195e+09 _runtime‚7060.901045619 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335224876017664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001652 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990008774752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10903672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3996845787648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1219621042184192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3995500390912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3988664174318 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1223611050958944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚348 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991659 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22972461 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5156915050496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1220148420296704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5156876326912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36793515 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36783528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36783528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1224145266084352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10903677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335224469202944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001652 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22972179 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36208137 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991659 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36208131 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330067960967168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36208131 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1220148420296704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36208137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0>%\! ‡N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36208131 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1220148420296704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36208137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330067592876032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1224145266084352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1223609680930542 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1220148394123264 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1220148394123264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12068507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1219621016756224 $ _timestamp‚1.784326807526195e+09 _runtime‚7060.901045619 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335224876017664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001652 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990008774752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10903672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3996845787648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1219621042184192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3995500390912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_ba÷ðKÓìtch_end/requested_bytes.small_pool.freed‚ 3988664174318 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1223611050958944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚348 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991659 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22972461 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5156915050496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1220148420296704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5156876326912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36793515 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36783528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36783528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1224145266084352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10903677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335224469202944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001652 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22972179 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36208137 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991659 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36208131 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330067960967168Ü–KW2‚- _wandbruntime‚7060  _runtime‚7060f13Fƒ‚ 0loss/FRt2/assignment_loss‚1.7519253094787481 2loss/SRqqt1/assignment_loss‚0.5500801119393289 1loss/SRqqt2/detection_loss‚0.9156606496404537 %loss/total_loss‚8.971295281945677 _step‚349 2loss/FBt2/assignment_loss‚0.021116799795269495 _runtime‚7060.987893943 2loss/SRqqt2/assignment_loss‚0.6352678756437606 /loss/FBt2/detection_loss‚0.9833386601181701 trainer/global_step‚4249 0loss/FRt1/assignment_loss‚0.7002050079185677 1loss/FBt1/assignment_loss‚0.03518192334671086 /loss/FRt1/detection_loss‚1.5457442056793136 /loss/FRt2/detection_loss‚0.6465128953556821 1loss/SRqqt1/detection_loss‚0.5959095337292257 .loss/FBt1/detection_loss‚0.590352309300446 epoch‚10 % _timestamp‚1.7843268076135657e+09-Ëcd „‡ 0loss/FRt1/assignment_loss‚0.7002050079185677 1loss/FBt1/assignment_loss‚0.03518192334671086 /loss/FRt1/detection_loss‚1.5457442056793136 /loss/FRt2/detection_loss‚0.6465128953556821 1loss/SRqqt1/detection_loss‚0.5959095337292257 .loss/FBt1/detection_loss‚0.590352309300446 epoch‚10 % _timestamp‚1.7843268076135657e+09 0loss/FRt2/assignment_loss‚1.7519253094787481 2loss/SRqqt1/assignment_loss‚0.5500801119393289 1loss/SRqqt2/detection_loss‚0.9156606496404537 %loss/total_loss‚8.971295281945677 _step‚349 2loss/FBt2/assignment_loss‚0.021116799795269495 _runtime‚7060.987893943 2loss/SRqqt2/assignment_loss‚0.6352678756437606 /loss/FBt2/detection_loss‚0.9833386601181701 trainer/global_step‚4249ÝÛ¬Áž…‚(:“ ›ÕêÒÔˆü® gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°¹åÈ'†‚(:œ ›ÕêÒ•ñŠ° disk.rbd2.out‚ 49.2021484375 disk.drbd2165.out‚0 disk.rbd1.in‚ 506.52734375) disk./.usagePercent‚39.20158889335275 disk.dm-0.in‚ 14585.3125 disk.dm-1.in‚ 0.8359375 disk.nvme0n1.in‚ 14585.3125 disk.sda.in‚ 0.8359375$ disk./.usageGB‚584.1320037841797 disk.sda3.in‚ 0.8359375) proc.memory.availableMB‚ 1893.96484375# proc.memory.rssMB‚ 2909.63671875 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚65488.986328125& disk.nvme0c0n1.out‚65488.986328125 disk.sda2.in‚0* proc.memory.percent‚17.759013175964355 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.out‚ 426.390625$ disk.nvme0n1.out‚65488.986328125 disk.sda2.out‚0 disk.sda.out‚ 426.390625 network.sent‚ 7.952487e+06 network.recv‚ 1.803396e+06 disk.dm-1.out‚ 426.390625# disk.rbd1.out‚17771.02099609375$ memory_percent‚88.44015598297119 cpu‚25.567723148174633 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 14585.3125b¨`”‡‚(:‰ ªÕêÒ䨷¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôXI´<ˆ‚(:± ªÕêÒÑàà® proc.cpu.threads‚17" disk.nvme0n1.in‚14586.26953125 network.recv‚ 1.808023e+06 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17771.14599609375 disk.sda2.in‚0# proc.memory.rssMB‚ 2909.63671875 network.sent‚ 8.015591e+06 disk.dm-1.out‚ 426.55859375( disk.nvme0c0n1.out‚65502.30126953125$ disk./.usageGB‚584.1057662963867 disk.sda.out‚ 426.55859375 disk.rbd0.in‚0 disk.dm-1.in‚ 0.8359375$ memory_percent‚88.44127655029297* proc.memory.percent‚17.759013175964355 cpu‚25.56787163525222 disk.rbd2.in‚0.015625# disk.dm-0.out‚65502.30126953125 disk.sda3.in‚ 0.8359375* disk./.usagePercent‚39.199828073532245 disk.sda.in‚ 0.8359375 disk.rbd0.out‚0 disk.dm-0.in‚14586.26953125 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚65502.30126953125 disk.rbd2.out‚ 49.2021484375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 426.55859375$ disk.nvme0c0n1.in‚14586.26953125& proc.memory.availableMB‚ 1893.78125λ}lŸ‰‚(:” ¹ÕêҤᶬ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BãR`)Š‚(:ž ¹ÕêÒý–ͯ disk.drbd2165.out‚0 disk.rbd2.out‚49.3125$ disk.nvme0c0n1.in‚14695.14453125) disk./.usagePercent‚39.19988132309937 disk.rbd1.in‚ 506.52734375# proc.memory.rssMB‚ 2909.63671875* proc.memory.percent‚17.759013175964355 network.recv‚ 1.811705e+06# disk./.usageGB‚584.106559753418 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 426.90625& proc.memory.availableMB‚ 1893.78125 disk.sda.in‚ 0.8359375 disk.sda2.out‚0" disk.dm-0.out‚65506.6787109375 disk.sda3.in‚ 0.8359375" disk.nvme0n1.in‚14695.14453125 disk.sda.out‚ 426.90625 disk.sda2.in‚0# disk.rbd1.out‚17771.14599609375 disk.rbd0.out‚0 disk.dm-0.in‚14695.14453125 cpu‚25.568319293825304 proc.cpu.threads‚17 network.sent‚ 8.020787e+06% disk.nvme0n1.out‚65506.6787109375 disk.dm-1.out‚ 426.90625' disk.nvme0c0n1.out‚65506.6787109375$ memory_percent‚88.44127655029297 disk.rbd0.in‚0 disk.sda1.in‚0ƒ0R”‹‚(:‰ ÈÕêÒ»ÈÍ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈÞ¾c/Œ‚(:¤ ÈÕêÒ„ݪ® network.sent‚ 8.026048e+06 disk.sda1.in‚0! disk.nvme0n1.in‚ 14695.9765625 proc.cpu.threads‚17 network.recv‚ 1.815383e+06# disk.rbd1.out‚17771.20849609375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚65525.474609375 disk.sda.in‚ 0.8359375) proc.memory.availableMB‚ 1894.12890625) disk./.usagePercent‚39.20030578359114 disk.rbd1.in‚ 506.52734375& disk.nvme0c0n1.out‚65525.474609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.out‚50.0625 disk.dm-1.out‚ 428.08984375# proc.memory.rssMB‚ 2909.63671875 disk.rbd0.in‚0! disk.dm-0.out‚65525.474609375 disk.sda3.out‚ 428.08984375# disk.nvme0c0n1.in‚ 14695.9765625$ memory_percent‚88.43915462493896 cpu‚25.56853554415538$ disk./.usageGB‚584.1128845214844 disk.dm-0.in‚ 14695.9765625 disk.sda1.out‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375* proc.memory.percent‚17.759013175964355 disk.rbd2.in‚0.015625 disk.sda.out‚ 428.08984375R”‚(:‰ ×ÕêÒÔ³µ¬ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª´E]5Ž‚(:ª ×ÕêÒå ˜¯" disk.nvme0n1.in‚14696.93359375& disk.nvme0n1.out‚65540.85595703125# memory_percent‚88.4328842163086 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚14696.93359375 disk.dm-1.in‚ 0.8359375 network.recv‚ 1.818883e+06 disk.sda3.out‚ 429.046875$ disk.nvme0c0n1.in‚14696.93359375 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17771.20849609375$ disk./.usageGB‚584.1121520996094 disk.sda2.in‚0 disk.sda1.in‚0 cpu‚25.568979771795526* disk./.usagePercent‚39.200256630144565 disk.rbd0.out‚0# disk.dm-0.out‚65540.85595703125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 50.205078125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚65540.85595703125* proc.memory.percent‚17.759013175964355 disk.drbd2165.in‚0 disk.sda.in‚ 0.8359375 disk.sda.out‚ 429.046875& proc.memory.availableMB‚ 1895.15625# proc.memory.rssMB‚ 2909.63671875 network.sent‚ 8.031681e+06 disk.sda1.out‚0 disk.dm-1.out‚ 429.046875 disk.sda3.in‚ 0.8359375œà\2- _wandbruntime‚7138  _runtime‚7138bncHÉà % _timestamp‚1.7843268852648277e+09 _runtime‚7138.639435693 _step‚350 ' lr-AdamW/pg1‚0.00024816588530199524 ' lr-AdamW/pg2‚0.00024816588530199524 trainer/global_step‚4299à”êÎ‘È trainer/global_step‚4299 % _timestamp‚1.7843268852648277e+09 _runtime‚7138.639435693 _step‚350 ' lr-AdamW/pg1‚0.00024816588530199524 ' lr-AdamW/pg2‚0.00024816588530199524Þü´º2’- _wandbruntime‚7138  _runtime‚7138•âÄÿ'“ùO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1237811321184768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821006 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1233770429898752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333864426852352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1233770429898752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36610225 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843268852659507e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1237811321184768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36610231 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1237272571736318 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1237812692747776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1233770456072192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12201603 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37200802 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4035322421502 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12201891 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11028279 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033977821180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4040891286016 _runtime‚7138.640802223 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339079019551232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚4299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36610231 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4040891286016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚06ñy^”þO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37210775 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37200802 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36610225 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4042236675584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1233770429898752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333864426852352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1233770429898752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36610225 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843268852659507e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1237811321184768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36610231 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1237272571736318 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1237812692747776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1233770456072192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12201603 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37200802 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4035322421502 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12201891 QIDeviceStatsi©PѦMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11028279 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033977821180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4040891286016 _runtime‚7138.640802223 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339079019551232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚4299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36610231 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4040891286016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1237811321184768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821006 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4036245887860 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4041816321536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339158088168960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339157681354240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1234048456802304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5215806817792 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1233515105665024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837704 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1234048430628864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5215768094208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11030827 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234048430628864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37219267 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36618437 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36618437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12204329 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1237551376980852 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37209280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843268866510785e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37209280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827711 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1238091618520576 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238091618520576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4043161718272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚7140.025924008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1234048456802304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4041816321536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11030822 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23235151 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4036245887860 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4041816321536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339158088168960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339157681354240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1234048456802304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5215806817792 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1233515105665024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837704 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1234048430628864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5215768094208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11030827 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234048430628864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37219267 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36618437 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36618437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12204329 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1237551376980852 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37209280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843268866510785e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37209280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827711 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1238091618520576 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238091618520576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4043161718272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚7140.025924008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1234048456802304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4041816321536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11030822 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23235151 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37020525 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚355 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251211527729936 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1247131338223616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1251757673620992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1251759045184000 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1247670492577792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343012933065216 _runtime‚7218.564756085 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5274222462464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247131312795648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4087207216640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11155429 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12338621 % _timestamp‚1.7843269651899743e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4088552606208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37020531 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1251757673620992 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337738747236352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1247670466404352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1247670466404352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4080214934288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647079 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1251759045184000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1251212897758226 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4081559534610 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚07\_(¬þO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37626554 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37020525 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4087207216640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343013337789952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37020531 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337738747236352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1247670466404352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1247670466404352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4080214934288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647079 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1251759045184000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1251212897758226 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4081559534610 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337739115327488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23493799 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5274185828864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37020525 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚355 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251211527729936 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1247131338223616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1251757673620992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1251759045184000 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1247670492577792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343012933065216 _runtime‚7218.564756085 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5274222462464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247131312795648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4087207216640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11155429 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12338621 % _timestamp‚1.7843269651899743e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4088552606208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37020531 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1251757673620992 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663763 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4088132252160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082483000968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23499229 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37635032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343091611631104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5275412516352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4088132252160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37635032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247948493307904 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673756 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37028737 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37028731 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843269665751338e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4089477648896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23498976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚356 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1247948467134464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337816601735168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4349 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1252036599386624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4081138400534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1252037970956800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12341004 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663763 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4088132252160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082483000968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23499229 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37635032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343091611631104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5275412516352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4088132252160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37635032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247948493307904 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673756 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37028737 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37028731 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843269665751338e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4089477648896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23498976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚356 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1247948467134464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337816601735168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4349 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1252036599386624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4081138400534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1252037970956800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12341004 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1265704026057216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4133523147264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37430831 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23757619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12474772 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341613435711488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4127796647718 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37430825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843270444570749e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38062279 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5333627483648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5333590850048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12475035 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚4399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1265704026057216 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4134868536832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1265151853751844 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4133523147264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346947063195136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341613067620352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38062279 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5333627483648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5333590850048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12475035 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚4399 A7DeviceStatsMonitor.on_train_baOatch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1265704026057216 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4134868536832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1265151853751844 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4133523147264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346947063195136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341613067620352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1265704026057216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4133523147264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37430831 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23757619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12474772çíÄ°2Ã- _wandbruntime‚7299  _runtime‚7299mÈe'ÄN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261848529813504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4135793579520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38070771 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1265984323393024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1265432029024668 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 $ _timestamp‚1.784327045842093e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1265982951822848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4134448182784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499815 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11285122 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499815 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4135793579520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4134448182784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261303283482624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347025733447168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261848529813504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11285127 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23762875 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4127375513642 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1261848503640064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12477492 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261303308910592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37439037 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1265430658996266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚4399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38060784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5334811328000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37439031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341690554028032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38070771 _runtime‚ 7299.21686179 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23762614 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341690922119168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12477748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509808 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37439037 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261848529813504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11285127 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23762875 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4127375513642 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1261848503640064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12477492 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261303308910592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37439037 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1265430658996266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚4399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38060784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5334811328000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37439031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341690554028032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38070771 _runtime‚ 7299.21686179 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23762614 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341690922119168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12477748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509808 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37439037 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4135793579520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38070771 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1265984323393024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1265432029024668 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 $ _timestamp‚1.784327045842093e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1265982951822848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4134448182784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499815 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568èq]´o2Æ- _wandbruntime‚7299  _runtime‚7299.pNêÇ 0loss/FRt1/assignment_loss‚0.7377770332301535 2loss/SRqqt1/assignment_loss‚0.5537231886568252 1loss/FBt1/assignment_loss‚0.02674948555506393 1loss/SRqqt1/detection_loss‚0.6344823596946938 .loss/FBt1/detection_loss‚0.688815110579133 trainer/global_step‚4399 3loss/FBt2/assignment_loss‚0.0034052174215901234 /loss/FBt2/detection_loss‚1.0183347910376532 epoch‚10 % _timestamp‚1.7843270459284587e+09 2loss/SRqqt2/assignment_loss‚0.5427773271166908 /loss/FRt2/detection_loss‚0.5413599626023401 1loss/SRqqt2/detection_loss‚0.9338928357396236 _runtime‚7299.302825585 _step‚361 .loss/FRt2/assignment_loss‚1.83028531315861 /loss/FRt1/detection_loss‚1.5025812639695841 %loss/total_loss‚9.014183888761963žW* Ȇ .loss/FRt2/assignment_loss‚1.83028531315861 /loss/FRt1/detection_loss‚1.5025812639695841 %loss/total_loss‚9.014183888761963 0loss/FRt1/assignment_loss‚0.7377770332301535 2loss/SRqqt1/assignment_loss‚0.5537231886568252 1loss/FBt1/assignment_loss‚0.02674948555506393 1loss/SRqqt1/detection_loss‚0.6344823596946938 .loss/FBt1/detection_loss‚0.688815110579133 trainer/global_step‚4399 3loss/FBt2/assignment_loss‚0.0034052174215901234 /loss/FBt2/detection_loss‚1.0183347910376532 epoch‚10 % _timestamp‚1.7843270459284587e+09 2loss/SRqqt2/assignment_loss‚0.5427773271166908 /loss/FRt2/detection_loss‚0.5413599626023401 1loss/SRqqt2/detection_loss‚0.9338928357396236 _runtime‚7299.302825585 _step‚361é7}Ë•É‚(:Š ‹×êҾؽ® gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0жî2Ê‚(:§ ‹×êÒýŒ­° disk.sda2.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚65637.3837890625 disk.nvme0n1.in‚ 14976.109375 disk.sda.in‚ 0.8359375 disk.sda.out‚ 433.45703125" disk.dm-0.out‚65637.3837890625" proc.memory.rssMB‚ 2912.2734375 network.recv‚ 1.880725e+06 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17771.65380859375 disk.dm-1.out‚ 433.45703125 disk.sda3.out‚ 433.45703125 disk.rbd2.out‚ 51.814453125 disk.dm-0.in‚ 14976.109375 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1892.1015625* disk./.usagePercent‚39.200400506378806$ disk./.usageGB‚584.1142959594727" disk.nvme0c0n1.in‚ 14976.109375) proc.memory.percent‚17.77510643005371 network.sent‚ 8.253621e+06% disk.nvme0n1.out‚65637.3837890625 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.8359375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 cpu‚25.57461043692024 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.8359375$ memory_percent‚88.45152854919434B- c5Ë‚(:ª š×êÒÚï¯ disk.rbd0.in‚0$ disk.nvme0c0n1.in‚14977.06640625) disk./.usagePercent‚39.20042508310209 disk.sda.in‚ 0.8359375 disk.rbd1.in‚ 506.52734375 disk.rbd2.out‚ 51.814453125 disk.sda3.out‚ 433.65234375 disk.rbd0.out‚0! disk.dm-0.out‚65648.587890625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2917.55078125 disk.rbd2.in‚0.015625 disk.dm-0.in‚14977.06640625 proc.cpu.threads‚17 network.sent‚ 8.316183e+06 disk.sda2.in‚0# disk.rbd1.out‚17771.65380859375 disk.drbd2165.out‚0 disk.dm-1.out‚ 433.65234375' proc.memory.availableMB‚ 1886.609375$ disk./.usageGB‚584.1146621704102" disk.nvme0n1.in‚14977.06640625 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375$ memory_percent‚88.48505020141602 network.recv‚ 1.886061e+06 disk.sda.out‚ 433.65234375 disk.sda1.out‚0& disk.nvme0c0n1.out‚65648.587890625 cpu‚25.58024820796411 disk.dm-1.in‚ 0.8359375* proc.memory.percent‚17.807316780090332$ disk.nvme0n1.out‚65648.587890625 disk.sda2.out‚0äE²Œ“Ì‚(:ˆ š×êÒÛ¢ÔÈ gpu.0.gpu‚25.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÃ2Í- _wandbruntime‚7326  _runtime‚7326ýØ„NWγ{ 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877983331680298 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5869525074958801 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.762055516242981 *jet/accuracy_1_of_1‚0.2606966495513916 6validation_average_jet_accuracy‚0.3098575472831726 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8077475428581238 :"PuN3äþb<rity/*FRt1SRqqt0FBt/event_purity‚0.30041858553886414 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4369643032550812 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.135749951004982 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17338302731513977 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45115017890930176 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6648880243301392 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6601904630661011 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01585846021771431 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09748972207307816 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4760957360267639 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 /particle/accuracy_4_of_4‚0.9405415058135986 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9834417700767517 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6730804443359375 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3059850335121155 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6950869560241699 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843270728998497e+09 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5673287510871887 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9926702380180359 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15708844363689423 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25290849804878235 +jet/accuracy_2_of_2‚0.12568069994449615 'particle/f_score‚0.6224669814109802 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15682590007781982 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012738369405269623 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17887823283672333 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.518957793712616 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1594153195619583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6708546876907349 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 jet/accuracy_5_of_5‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09748972207307816 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5776341557502747 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7666856050491333 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7980347871780396 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.541538417339325 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5570833683013916 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5647763013839722 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5458027720451355 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8925355076789856 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934741258621216 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5445842742919922 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6296176910400391 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6639004945755005 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23698069155216217 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2258087545633316 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9250911474227905 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8566670417785645 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.563403308391571 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17223107814788818 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023435717448592186 (particle/accuracy‚0.8390145301818848 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19260774552822113 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_2‚0.7171148061752319 *jet/accuracy_2_of_3‚0.4885481894016266 jet/accuracy_4_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5812041163444519 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8642123937606812 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9206590056419373 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5123718976974487 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3166978359222412 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11900628358125687 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15397295355796814 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887983202934265 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909548163414001 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17214082181453705 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 +particle/sensitivity‚0.6180226802825928 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.29718291759490967 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834891200065613 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17168208956718445 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.035815756767988205 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16071908175945282 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841458201408386 +jet/accuracy_3_of_4‚0.08842239528894424 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08447064459323883 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20133186876773834 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5606001019477844 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7565397620201111 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5943344831466675 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8027588725090027 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.512579619884491 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚4399 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15082554519176483 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09007883071899414 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6699397563934326 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967266917228699 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.49084222316741943 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030221223831176758 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5953347682952881 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.762055516242981 particle/accuracy_1_of_4‚1 _runtime‚7326.275090034 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7002826929092407 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954648613929749 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15708844363689423 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897609949111938 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955620765686035 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932959675788879 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.512579619884491 /particle/accuracy_3_of_4‚0.9936224818229675 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3063727617263794 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1331821233034134 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944719672203064 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17523612082004547 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1688138097524643 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.785301923751831 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924536347389221 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6283761858940125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8098261952400208 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1782667338848114 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.654804000787582 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.559367299079895 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5217840075492859 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19090807437896729 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07043038308620453 epoch‚10 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953017234802246 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13013382256031036 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006727621890604496 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09009486436843872 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9866725206375122 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6563354730606079 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8792030811309814 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.977596640586853 *jet/accuracy_2_of_4‚0.7392593026161194 _step‚362 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6610807776451111 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6096311807632446 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5260454416275024 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.605254054069519 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4660695493221283 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7294886112213135 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9833332896232605 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08439787477254868 particle/accuracy_2_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18711566925048828 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5513761043548584 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6254434585571289 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9720653295516968 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25309881567955017 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21207155287265778 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953669309616089 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.8994459509849548 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.34375ÅÒ¾=ϸ{ ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954648613929749 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15708844363689423 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897609949111938 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955620765686035 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932959675788879 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.512579619884491 /particle/accuracy_3_of_4‚0.9936224818229675 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3063727617263794 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1331821233034134 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944719672203064 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17523612082004547 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1688138097524643 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.785301923751831 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924536347389221 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6283761858940125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8098261952400208 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1782667338848114 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.654804000787582 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.559367299079895 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5217840075492859 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19090807437896729 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07043038308620453 epoch‚10 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953017234802246 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13013382256031036 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006727621890604496 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09009486436843872 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9866725206375122 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6563354730606079 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8792030811309814 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.977596640586853 *jet/accuracy_2_of_4‚0.7392593026161194 _step‚362 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6610807776451111 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6096311807632446 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5260454416275024 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.605254054069519 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4660695493221283 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7294886112213135 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9833332896232605 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08439787477254868 particle/accuracy_2_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18711566925048828 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5513761043548584 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6254434585571289 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9720653295516968 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25309881567955017 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21207155287265778 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953669309616089 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.8994459509849548 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.34375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877983331680298 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5869525074958801 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.762055516242981 *jet/accuracy_1_of_1‚0.2606966495513916 6validation_average_jet_accuracy‚0.3098575472831726 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8077475428581238 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30041858553886414 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4369643032550812 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.135749951004982 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17338302731513977 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45115017890930176 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6648880243301392 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6601904630661011 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01585846021771431 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09748972207307816 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4760957360267639 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 /particle/accuracy_4_of_4‚0.9405415058135986 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9834417700767517 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6730804443359375 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3059850335121155 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6950869560241699 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843270728998497e+09 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5673287510871887 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9926702380180359 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15708844363689423 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25290849804878235 +jet/accuracy_2_of_2‚0.12568069994449615 'particle/f_score‚0.6224669814109802 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15682590007781982 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012738369405269623 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17887823283672333 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.518957793712616 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1594153195619583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6708546876907349 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 jet/accuracy_5_of_5‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09748972207307816 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5776341557502747 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7666856050491333 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7980347871780396 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.541538417339325 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5570833683013916 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5647763013839722 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5458027720451355 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8925355076789856 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934741258621216 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5445842742919922 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6296176910400391 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6639004945755005 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23698069155216217 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2258087545633316 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9250911474227905 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8566670417785645 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.563403308391571 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17223107814788818 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023435717448592186 (particle/accuracy‚0.8390145301818848 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19260774552822113 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_2‚0.7171148061752319 *jet/accuracy_2_of_3‚0.4885481894016266 jet/accuracy_4_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5812041163444519 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8642123937606812 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9206590056419373 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5123718976974487 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3166978359222412 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11900628358125687 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15397295355796814 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887983202934265 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909548163414001 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17214082181453705 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 +particle/sensitivity‚0.6180226802825928 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.29718291759490967 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834891200065613 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17168208956718445 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.035815756767988205 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16071908175945282 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841458201408386 +jet/accuracy_3_of_4‚0.08842239528894424 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08447064459323883 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20133186876773834 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5606001019477844 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7565397620201111 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5943344831466675 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8027588725090027 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.512579619884491 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚4399 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15082554519176483 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09007883071899414 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6699397563934326 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967266917228699 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.49084222316741943 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030221223831176758 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5953347682952881 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.762055516242981 particle/accuracy_1_of_4‚1 _runtime‚7326.275090034 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7002826929092407 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548êŠä`”Ђ(:‰ ©×êÒñ¤Ç¬ gpu.0.gpu‚72.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð©‰A0Ñ‚(:« ©×êÒ—Þ¿®* proc.memory.percent‚17.823433876037598 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.out‚ 433.92578125 disk.sda.out‚ 433.92578125 disk.sda2.in‚0) disk./.usagePercent‚39.20041817089867" disk.nvme0c0n1.in‚ 15070.640625 disk.nvme0n1.in‚ 15070.640625 disk.rbd1.in‚ 506.52734375 network.sent‚ 8.321242e+06 disk.rbd2.out‚ 51.892578125( disk.nvme0c0n1.out‚65656.87548828125 proc.cpu.threads‚17 network.recv‚ 1.889604e+06# disk.dm-0.out‚65656.87548828125 disk.dm-1.in‚ 0.8359375& disk.nvme0n1.out‚65656.87548828125 disk.sda.in‚ 0.8359375$ memory_percent‚88.50150108337402 disk.sda1.out‚0 disk.dm-0.in‚ 15070.640625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.8359375# disk.rbd1.out‚17771.73193359375( proc.memory.availableMB‚ 1883.9140625# proc.memory.rssMB‚ 2920.19140625 cpu‚25.582777789546682# disk./.usageGB‚584.114559173584 disk.dm-1.out‚ 433.92578125 disk.sda2.oz4ÎGut‚0)'¦èŸÒ‚(:” ¸×êÒª¢Þ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þƒñ/Ó‚(:¤ ¸×êÒȸí­ disk.nvme0n1.in‚ 15070.640625 disk.sda.out‚ 434.15625 disk.rbd1.in‚ 506.52734375 disk.dm-0.in‚ 15070.640625( disk.nvme0c0n1.out‚65663.61962890625 disk.dm-1.in‚ 0.8359375$ memory_percent‚88.50042819976807# proc.memory.rssMB‚ 2920.19140625 proc.cpu.threads‚17 disk.sda3.out‚ 434.15625) disk./.usagePercent‚39.20040229843155 disk.sda2.out‚0# disk.dm-0.out‚65663.61962890625 disk.sda3.in‚ 0.8359375) proc.memory.availableMB‚ 1884.08984375& disk.nvme0n1.out‚65663.61962890625 disk.sda.in‚ 0.8359375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625* proc.memory.percent‚17.823433876037598 network.recv‚ 1.894451e+06$ disk./.usageGB‚584.1143226623535" disk.nvme0c0n1.in‚ 15070.640625 disk.sda1.out‚0# disk.rbd1.out‚17771.73193359375 disk.rbd2.out‚ 52.701171875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 434.15625 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.582919224912125 network.sent‚ 8.378036e+06´ƒDŸ”Ô‚(:‰ Ç×êÒÝÆÁ¬ gpu.0.gpu‚56.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø™,9Õ‚(:® Ç×êÒꋯ( disk.nvme0c0n1.out‚65671.26416015625 disk.rbd0.out‚0 disk.dm-0.in‚15071.59765625& disk.nvme0n1.out‚65671.26416015625 disk.sda.in‚ 0.8359375$ memory_percent‚88.50078582763672# disk.dm-0.out‚65671.26416015625" disk.nvme0n1.in‚15071.59765625& proc.memory.availableMB‚ 1884.03125 cpu‚25.583283481222402 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 8.383639e+06# disk.rbd1.out‚17771.79443359375 disk.dm-1.out‚ 434.8203125 disk.sda2.out‚0# proc.memory.rssMB‚ 2920.19140625 disk.drbd2165.in‚0 disk.sda3.in‚ 0.8359375$ disk.nvme0c0n1.in‚15071.59765625 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚584.1144180297852 disk.rbd1.in‚ 506.52734375 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 434.8203125* proc.memory.percent‚17.823433876037598 disk.rbd2.out‚52.86767578125 disk.sda.out‚ 434.8203125 network.recv‚ 1.898058e+06( disk./.usagePercent‚39.2004086986199 disk.drbd2165.out‚0è:Ú”Ö‚(:‰ Ö×êÒÃʪ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«kÌÝ7ׂ(:¬ Ö×êÒ¥˜ì¯$ memory_percent‚88.50090503692627) proc.memory.availableMB‚ 1884.01171875 disk.dm-0.in‚ 15072.4296875# proc.memory.rssMB‚ 2920.19140625) disk./.usagePercent‚39.20040818660483$ disk./.usageGB‚584.1144104003906# disk.nvme0c0n1.in‚ 15072.4296875 disk.sda.out‚ 435.203125 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17771.79443359375 disk.rbd0.in‚0 disk.rbd2.out‚52.86767578125 disk.drbd2165.out‚0 disk.rbd2.in‚0.015625 cpu‚25.583548042539213 network.recv‚ 1.901368e+06( disk.nvme0c0n1.out‚65682.73876953125! disk.nvme0n1.in‚ 15072.4296875# disk.dm-0.out‚65682.73876953125 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 8.388204e+06 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 435.203125* proc.memory.percent‚17.823433876037598 disk.sda.in‚ 0.8359375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.8359375& disk.nvme0n1.out‚65682.73876953125 disk.dm-1.out‚ 435.203125jaϞ؂(:“ å×êÒé‘£¬ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.587' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.91739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎØ} 6Ù‚(:« å×êÒ–÷Ó®$ memory_percent‚88.50111961364746$ disk./.usageGB‚584.1145133972168! disk.nvme0c0n1.in‚ 15082.28125 disk.rbd0.in‚0 disk.dm-0.in‚ 15082.28125 disk.dm-1.out‚ 435.37109375 cpu‚25.583926713877453 network.sent‚ 8.393471e+06 disk.sda3.out‚ 435.37109375& disk.nvme0n1.out‚65687.17822265625( proc.memory.availableMB‚ 1883.9765625 disk.drbd2165.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17771.93505859375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.8359375 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 0.8359375 network.recv‚ 1.905272e+06 disk.sda2.out‚0# disk.dm-0.out‚65687.17822265625 disk.sda1.out‚0# proc.memory.rssMB‚ 2920.19140625 proc.cpu.threads‚17) disk./.usagePercent‚39.20041509880826 disk.sda.in‚ 0.8359375 disk.sda2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.823433876037598( disk.nvme0c0n1.out‚65687.17822265625 disk.nvme0n1.in‚ 15082.28125 disk.sda.out‚ 435.37109375 disk.rbd2.out‚54.439941406257iÓ2Ú- _wandbruntime‚7404  _runtime‚7404’¹áÇÛÁ & lr-AdamW/pg2‚0.0002462063811031915 trainer/global_step‚4449 % _timestamp‚1.7843271507557652e+09 _runtime‚7404.130373793 _step‚363 & lr-AdamW/pg1‚0.0002462063811031915s¤‚ÌÜÆ & lr-AdamW/pg1‚0.0002462063811031915 & lr-AdamW/pg2‚0.0002462063811031915 trainer/global_step‚4449 % _timestamp‚1.7843271507557652e+09 _runtime‚7404.130373793 _step‚363ëåöÉ2Ý- _wandbruntime‚7404  _runtime‚7404ÈÚ‘Óþ'ÞøO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187027079168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5401041999360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1279298520850432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1283485574103040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11418647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37973137 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1279298547023872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37973131 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4449 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1279298520850432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784327150756881e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1283485574103040 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279298547023872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4185681689600 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37973137 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5401074438656 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚364 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1283484202540032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520150 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4178518291941 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4185681689600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350936542250496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1283484202540032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38547013 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1278745399545856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38547013 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11418647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37973137 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1279298547023872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37973131 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4449 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1279298520850432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784327150756881e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1283485574103040 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279298547023872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4185681689600 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37973137 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5401074438656 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚364 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1283484202540032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520150 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4178518291941 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4185681689600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350936542250496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1283484202540032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38547013 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1278745399545856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38547013 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4180786358621 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536848 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526855 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1283204093110621 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1279023281324032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4179441758187 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37981337 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚365 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526855 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5402249682432 _runtime‚7405.518361842 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12650287 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351015636864512 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚4449 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24071237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11421195 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345612986658816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5402282114560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38545518 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1283763128305664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4187952121856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1279023306752000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1283764499875840 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1283764499875840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38555505 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1283202723082219 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4186606725120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37981337 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351015236341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚379813437Qøò 'â‡N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1283764499875840 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1283764499875840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38555505 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1283202723082219 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4186606725120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37981337 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351015236341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37981343 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38545518 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4180786358621 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536848 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526855 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1283204093110621 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1279023281324032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4179441758187 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37981337 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚365 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526855 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5402249682432 _runtime‚7405.518361842 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12650287 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351015636864512 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚4449 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24071237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11421195 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345612986658816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5402282114560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38545518 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1283763128305664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4187952121856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1279023306752000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175í2ã- _wandbruntime‚7405  _runtime‚7405¸M?ä‚ _runtime‚7405.604436415 _step‚366 /loss/FBt2/detection_loss‚0.6587387536488035 2loss/SRqqt2/assignment_loss‚0.5465467904730759 1loss/FBt2/assignment_loss‚0.06716690411786216 /loss/FRt2/detection_loss‚0.5997667695410476 % _timestamp‚1.7843271522300913e+09 2loss/SRqqt1/assignment_loss‚0.5293104785644233 2loss/FBt1/assignment_loss‚0.007240712619533499 $loss/total_loss‚8.58886965177623 epoch‚11 trainer/global_step‚4449 0loss/FRt2/assignment_loss‚1.8074283979360857 /loss/FRt1/detection_loss‚1.4768644012556542 1loss/SRqqt1/detection_loss‚0.6897112370164897 0loss/SRqqt2/detection_loss‚0.958528334274888 0loss/FBt1/detection_loss‚0.49126342559028546 0loss/FRt1/assignment_loss‚0.7563034467380815r²^ å‡ _runtime‚7405.604436415 _step‚366 /loss/FBt2/detection_loss‚0.6587387536488035 2loss/SRqqt2/assignment_loss‚0.5465467904730759 1loss/FBt2/assignment_loss‚0.06716690411786216 /loss/FRt2/detection_loss‚0.5997667695410476 % _timestamp‚1.7843271522300913e+09 2loss/SRqqt1/assignment_loss‚0.5293104785644233 2loss/FBt1/assignment_loss‚0.007240712619533499 $loss/total_loss‚8.58886965177623 epoch‚11 trainer/global_step‚4449 0loss/FRt2/assignment_loss‚1.8074283979360857 /loss/FRt1/detection_loss‚1.4768644012556542 1loss/SRqqt1/detection_loss‚0.6897112370164897 0loss/SRqqt2/detection_loss‚0.958528334274888 0loss/FBt1/detection_loss‚0.49126342559028546 0loss/FRt1/assignment_loss‚0.7563034467380815îtÇ+”æ‚(:‰ ô×êÒù›È® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®5å3ç‚(:¨ ô×êҩ$ disk.nvme0c0n1.in‚15130.55078125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.8359375) proc.memory.availableMB‚ 1883.61328125 disk.sda.in‚ 0.8359375 disk.sda.out‚435.6875 disk.sda3.out‚435.6875& disk.nvme0n1.out‚65699.13720703125 disk.dm-0.in‚15130.55078125 network.recv‚ 1.908995e+06 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚17771.93505859375 disk.dm-1.out‚435.6875 disk.dm-1.in‚ 0.8359375# proc.memory.rssMB‚ 2920.19140625* proc.memory.percent‚17.823433876037598 cpu‚25.584157735221822 disk.rbd2.out‚54.43994140625# disk.dm-0.out‚65699.13720703125 network.sent‚ 8.398534e+06) disk./.usagePercent‚39.20042021895894$ disk./.usageGB‚584.1145896911621 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚15130.55078125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0( disk.nvme0c0n1.out‚65699.13720703125# memory_percent‚88.5033369064331y[×Lžè‚(:“ ƒØêÒÔûô« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚322.848' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.56960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ²a8é‚(:­ ƒØêÒÀÃí® network.recv‚ 1.915272e+06 disk.sda3.in‚ 0.8359375$ disk.nvme0n1.out‚65704.396484375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd2.out‚ 54.5107421875$ disk.nvme0c0n1.in‚15130.55078125 disk.sda.out‚ 436.65234375 disk.rbd1.in‚ 506.52734375 disk.sda1.out‚0 network.sent‚ 8.46167e+06$ disk./.usageGB‚584.1116256713867 disk.sda3.out‚ 436.65234375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625* proc.memory.percent‚17.823433876037598" disk.nvme0n1.in‚15130.55078125 disk.sda.in‚ 0.8359375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 436.65234375 cpu‚25.584493434507102) disk./.usagePercent‚39.20022130110484& disk.nvme0c0n1.out‚65704.396484375# disk.rbd1.out‚17772.06005859375! disk.dm-0.out‚65704.396484375$ memory_percent‚88.50185871124268) proc.memory.availableMB‚ 1883.85546875# proc.memory.rssMB‚ 2920.19140625 disk.sda2.in‚0 disk.dm-0.in‚15130.55078125 disk.drbd2165.out‚0 proc.cpu.threads‚17¹Ùdžê‚(:“ ’ØêÒј֬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-S­7ë‚(:¬ ’ØêÒ”õ® network.sent‚ 8.467267e+06 disk.sda.in‚ 0.8359375 disk.sda2.out‚0* proc.memory.percent‚17.823433876037598$ disk./.usageGB‚584.1115989685059& disk.nvme0n1.out‚65721.39697265625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚88.50188255310059# proc.memory.rssMB‚ 2920.19140625 proc.cpu.threads‚17 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.dm-0.in‚ 15131.3828125# disk.dm-0.out‚65721.39697265625( disk.nvme0c0n1.out‚65721.39697265625 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 437.8828125 disk.drbd2165.in‚0 disk.sda3.out‚ 437.8828125 disk.sda.out‚ 437.8828125# disk.rbd1.out‚17772.06005859375# disk.nvme0c0n1.in‚ 15131.3828125( proc.memory.availableMB‚ 1883.8515625 cpu‚25.584691559392283! disk.nvme0n1.in‚ 15131.3828125 network.recv‚ 1.918416e+06( disk./.usagePercent‚39.2002195090521 disk.sda3.in‚ 0.8359375 disk.sda2.in‚0 disk.rbd2.out‚ 54.5107421875EÁh—ì‚(:… ¡ØêÒ¬Ðû« gpu.0.gpu‚7.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚231.66' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/èÛ<í‚(:± ¡ØêÒ®ÜÚ® disk.sda1.in‚0 network.sent‚ 8.471798e+06 network.recv‚ 1.921843e+06$ disk.nvme0c0n1.in‚15132.33984375# proc.memory.rssMB‚ 2920.19140625* proc.memory.percent‚17.823433876037598 cpu‚25.58509138432115 disk.drbd2165.in‚0 disk.dm-1.out‚ 438.34765625 disk.sda.in‚ 0.8359375 disk.sda.out‚ 438.34765625 disk.rbd0.out‚0 disk.rbd2.out‚54.60205078125 disk.drbd2165.out‚0& proc.memory.availableMB‚ 1883.78125 disk.sda3.in‚ 0.8359375$ memory_percent‚88.50231170654297) disk./.usagePercent‚39.20022104509731 disk.sda3.out‚ 438.34765625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0$ disk./.usageGB‚584.1116218566895& disk.nvme0n1.out‚65730.08837890625 disk.sda2.out‚0" disk.nvme0n1.in‚15132.33984375 disk.dm-1.in‚ 0.8359375( disk.nvme0c0n1.out‚65730.08837890625 disk.rbd1.in‚ 506.52734375 disk.dm-0.in‚15132.33984375# disk.dm-0.out‚65730.08837890625 proc.cpu.threads‚17# disk.rbd1.out‚17772.13818359375x,9xžî‚(:“ °ØêÒôùè« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.579' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.31580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðçm+ï‚(:  °ØêÒäÓ„¯ disk.drbd2165.out‚0$ disk./.usageGB‚584.1116142272949# proc.memory.rssMB‚ 2920.19140625 proc.cpu.threads‚17# disk.dm-0.out‚65736.58740234375 disk.nvme0n1.in‚ 15425.1875 disk.sda2.in‚0* proc.memory.percent‚17.823433876037598) disk./.usagePercent‚39.20022053308224 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.8359375 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 network.sent‚ 8.477396e+06 network.recv‚ 1.925126e+06 disk.nvme0c0n1.in‚ 15425.1875( disk.nvme0c0n1.out‚65736.58740234375 disk.sda.in‚ 0.8359375 disk.rbd0.out‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 438.84375& disk.nvme0n1.out‚65736.58740234375 cpu‚25.585285708905772 disk.rbd2.out‚54.60205078125 disk.dm-0.in‚ 15425.1875 disk.dm-1.out‚ 438.84375 disk.sda.out‚ 438.84375# disk.rbd1.out‚17772.13818359375 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚88.50381374359131) proc.memory.availableMB‚ 1883.53515625’ JÏ2ð- _wandbruntime‚7483  _runtime‚7483-‹BÇñÁ & lr-AdamW/pg1‚0.0002455391326687378 & lr-AdamW/pg2‚0.0002455391326687378 trainer/global_step‚4499 % _timestamp‚1.7843272297967129e+09 _runtime‚7483.171384183 _step‚367¥Ž9ÌòÆ & lr-AdamW/pg1‚0.0002455391326687378 & lr-AdamW/pg2‚0.0002455391326687378 trainer/global_step‚4499 % _timestamp‚1.7843272297967129e+09 _runtime‚7483.171384183 _step‚367ïæ{Ó…2ó- _wandbruntime‚7483  _runtime‚7483»_Q’9"ôùO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1297431926539264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38383431 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24331594 _step‚368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4233343009792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38962792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1297431926539264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38383431 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12785797 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38962792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354870797917696 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1296864243859705 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚4499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38972765 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4224755405049 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231997620224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24331868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346223 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4231997620224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38383437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12786066 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1296865613887995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38383437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4233343009792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5460977282560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚7483.172641818 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1293198583529472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356202 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354871202642432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11545797 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38972765 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11545802 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349409820635136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292639488454656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293198557356032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1297430554976256 % _timestamp‚1.7843272297978916e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293198583529472 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293198557356032 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1297430554976256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349410188726272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292639488454656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293198557356032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1297430554976256 % _timestamp‚1.7843272297978916e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293198583529472 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293198557356032 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1297430554976256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349410188726272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1297431926539264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38383431 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12785797 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38962792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354870797917696 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1296864243859705 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚4499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38972765 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4224755405049 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231997620224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24331868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346223 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4231997620224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38383437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12786066 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1296865613887995ðJÑX%2ö- _wandbruntime‚7484  _runtime‚7484$zƒâ '÷ƒN UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362907 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1293476558086144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1297709480741888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4227023471729 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4499 % _timestamp‚1.7843272311835866e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372900 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38981257 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38391637 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1297144419132529 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372900 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4234268052480 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232922655744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349487675133952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38981257 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38971270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚369 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24336863 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349487307042816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12788523 _runtime‚7484.558400051 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4225678871295 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1293476584259584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1297710852312064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38391637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11548340 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354949473148416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1297143049104127 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362907 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1292917395660800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4232922655744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12788786 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1293476558086144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38971270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11548345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354949877865984múÜ^'øˆN D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12788523 _runtime‚7484.558400051 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4225678871295 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1293476584259584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1297710852312064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38391637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11548340 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354949473148416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1297143049104127 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362907 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1292917395660800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4232922655744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12788786 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1293476558086144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38971270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11548345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354949877865984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362907 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1293476558086144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1297709480741888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4227023471729 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4499 % _timestamp‚1.7843272311835866e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372900 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38981257 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38391637 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1297144419132529 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372900 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4234268052480 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232922655744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349487675133952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38981257 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38971270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚369 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24336863 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349487307042816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432ñ¯}Õž2ù- _wandbruntime‚7484  _runtime‚7484K˜CËú‚ /loss/FBt2/detection_loss‚0.8384974049113225 _runtime‚ 7484.64393534 _step‚370 0loss/FRt2/assignment_loss‚1.7310054763206182 2loss/SRqqt1/assignment_loss‚0.5052423996601251 2loss/FBt1/assignment_loss‚0.013255484273622595 /loss/FBt1/detection_loss‚0.4381408246094361 trainer/global_step‚4499 0loss/FRt1/assignment_loss‚0.6819775718715766 1loss/FBt2/assignment_loss‚0.07393170278971439 %loss/total_loss‚8.685895491254596 % _timestamp‚1.7843272312695942e+09 2loss/SRqqt2/assignment_loss‚0.6142334489555961 /loss/FRt1/detection_loss‚1.4361727038437644 /loss/FRt2/detection_loss‚0.6527901091859474 1loss/SRqqt2/detection_loss‚1.0552009328670215 epoch‚11 1loss/SRqqt1/detection_loss‚0.6454474319658509`ô¹ï{û‡ 1loss/SRqqt2/detection_loss‚1.0552009328670215 epoch‚11 1loss/SRqqt1/detection_loss‚0.6454474319658509 ItŠ£’/loss/FBt2/detection_loss‚0.8384974049113225 _runtime‚ 7484.64393534 _step‚370 0loss/FRt2/assignment_loss‚1.7310054763206182 2loss/SRqqt1/assignment_loss‚0.5052423996601251 2loss/FBt1/assignment_loss‚0.013255484273622595 /loss/FBt1/detection_loss‚0.4381408246094361 trainer/global_step‚4499 0loss/FRt1/assignment_loss‚0.6819775718715766 1loss/FBt2/assignment_loss‚0.07393170278971439 %loss/total_loss‚8.685895491254596 % _timestamp‚1.7843272312695942e+09 2loss/SRqqt2/assignment_loss‚0.6142334489555961 /loss/FRt1/detection_loss‚1.4361727038437644 /loss/FRt2/detection_loss‚0.6527901091859474òp:Q ‘ü‚(:† ¿Øêҹɵ¬ gpu.0.gpu‚28.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œñã7ý‚(:¬ ¿ØêÒԷ˯ disk.sda3.out‚ 440.09765625! disk.dm-0.out‚65744.142578125 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚65744.142578125$ disk.nvme0n1.out‚65744.142578125 disk.sda2.in‚0 network.sent‚ 8.481975e+06 network.recv‚ 1.928726e+06 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.823433876037598 disk.sda1.in‚0" disk.nvme0n1.in‚15426.01953125 disk.sda.in‚ 0.8359375) disk./.usagePercent‚39.20011531398566$ disk.nvme0c0n1.in‚15426.01953125# proc.memory.rssMB‚ 2920.19140625 cpu‚25.58561078546606 disk.rbd0.out‚0 disk.dm-1.in‚ 0.8359375 disk.rbd1.in‚ 506.52734375 disk.sda3.in‚ 0.8359375# disk.rbd1.out‚17772.20068359375$ memory_percent‚88.49968910217285( proc.memory.availableMB‚ 1884.2109375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 54.6650390625 disk.dm-0.in‚15426.01953125 disk.sda.out‚ 440.09765625 proc.cpu.threads‚17$ disk./.usageGB‚584.1100463867188 disk.dm-1.out‚ 440.09765625c9-†”þ‚(:‰ ÎØêÒ攸¬ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)PŸ5ÿ‚(:ª ÎØêÒ©Éï­' disk.nvme0c0n1.out‚65757.0732421875 disk.rbd0.out‚0 disk.rbd2.out‚ 54.6650390625" disk.dm-0.out‚65757.0732421875# disk.nvme0c0n1.in‚ 15426.9765625 disk.sda.in‚ 0.8359375# disk.rbd1.out‚17772.20068359375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 440.5859375 cpu‚25.585784534198925 proc.cpu.threads‚17 network.sent‚ 8.545162e+06! disk.nvme0n1.in‚ 15426.9765625 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 15426.9765625 disk.sda1.out‚0 disk.sda3.in‚ 0.8359375$ memory_percent‚88.50255012512207# proc.memory.rssMB‚ 2920.19140625 network.recv‚ 1.93376e+06 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1883.7421875 disk.sda.out‚ 440.5859375 disk.rbd0.in‚0% disk.nvme0n1.out‚65757.0732421875* proc.memory.percent‚17.823433876037598* disk./.usagePercent‚39.200068976621964$ disk./.usageGB‚584.1093559265137 disk.drbd2165.out‚0 disk.dm-1.out‚ 440.5859375cžª6”€‚(:‰ ÝØêÒØÙȬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ôó<‚(:± ÝØêÒèðó®( disk.nvme0c0n1.out‚65760.12060546875# disk.rbd1.out‚17772.27880859375 cpu‚25.586137026813056 disk.sda3.in‚ 0.8359375# disk.nvme0c0n1.in‚ 15426.9765625! disk.nvme0n1.in‚ 15426.9765625 disk.rbd2.out‚54.72802734375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2920.19140625 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1883.94921875 network.sent‚ 8.55022e+06) disk./.usagePercent‚39.20005848031306 disk.sda.in‚ 0.8359375 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.dm-1.out‚ 441.36328125 network.recv‚ 1.937158e+06 disk.sda3.out‚ 441.36328125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚88.50128650665283 disk.dm-0.in‚ 15426.9765625 disk.sda.out‚ 441.36328125 disk.sda2.out‚0# disk.dm-0.out‚65760.12060546875 disk.dm-1.in‚ 0.8359375$ disk./.usageGB‚584.1091995239258& disk.nvme0n1.out‚65760.12060546875 disk.rbd0.out‚0* proc.memory.percent‚17.823433876037598 proc.cpu.threads‚17Èþ(ˆ”‚‚(:‰ ìØêÒÐûæ« gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÎRv$ƒ‚(:™ ìØêÒ¬Ùõ­ disk.rbd2.out‚54.72802734375 disk.drbd2165.in‚0% disk.nvme0n1.out‚65769.5791015625 disk.sda.in‚ 0.8359375 disk.sda.out‚441.875 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.sda3.out‚441.875$ memory_percent‚88.50007057189941 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 15597.84375( proc.memory.availableMB‚ 1884.1484375 proc.cpu.threads‚17) disk./.usagePercent‚39.19999422242197 disk.rbd1.in‚ 506.52734375 disk.dm-0.in‚ 15597.84375 disk.sda1.out‚0* proc.memory.percent‚17.823433876037598" disk.dm-0.out‚65769.5791015625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.8359375' disk.nvme0c0n1.out‚65769.5791015625 cpu‚25.586403217134343 network.sent‚ 8.555323e+06$ disk./.usageGB‚584.1082420349121 disk.sda2.in‚0# disk.rbd1.out‚17772.27880859375 disk.sda1.in‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚441.875# proc.memory.rssMB‚ 2920.19140625 disk.nvme0n1.in‚ 15597.84375 network.recv‚ 1.940314e+06Îmei”„‚(:‰ ûØêÒç…œ¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚151.376' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’bˆ8…‚(:­ ûØêÒÀœ¸® disk.drbd2165.in‚0# proc.memory.rssMB‚ 2920.19140625 cpu‚25.58685587536024) disk./.usagePercent‚39.19998961428635 disk.sda2.in‚0" disk.dm-0.out‚65777.0361328125 disk.dm-1.out‚ 442.5546875 disk.sda3.in‚ 0.8359375 network.sent‚ 8.560922e+06$ disk.nvme0c0n1.in‚15598.80078125' disk.nvme0c0n1.out‚65777.0361328125 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.dm-1.in‚ 0.8359375# disk.rbd1.out‚17772.34130859375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625# memory_percent‚88.5016679763794 network.recv‚ 1.944133e+06% disk.nvme0n1.out‚65777.0361328125 disk.sda.out‚ 442.5546875 disk.rbd2.out‚54.72802734375 proc.cpu.threads‚17" disk.nvme0n1.in‚15598.80078125 disk.sda.in‚ 0.8359375 disk.sda3.out‚ 442.5546875 disk.sda2.out‚0) proc.memory.availableMB‚ 1883.88671875* proc.memory.percent‚17.823433876037598 disk.drbd2165.out‚0$ disk./.usageGB‚584.1081733703613 disk.dm-0.in‚15598.80078125 disk.sda1.in‚0 disk.sda1.out‚0Æ┆‚(:‰ ŠÙêÒâðÒ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚257.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÃ5A‡‚(:¶ ŠÙêÒ»ˆ¾® network.recv‚ 1.947755e+06 disk.sda.in‚ 0.8359375 disk.rbd2.in‚0.015625 disk.dm-0.in‚15598.80078125$ memory_percent‚88.50181102752686 disk.sda3.out‚ 443.01953125 disk.dm-1.in‚ 0.8359375" disk.nvme0n1.in‚15598.80078125 disk.sda2.out‚0# proc.memory.rssMB‚ 2920.19140625 disk.sda1.out‚0 disk.sda.out‚ 443.01953125# disk.dm-0.out‚65782.28564453125 disk.drbd2165.in‚0 disk.dm-1.out‚ 443.01953125 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.rbd2.out‚54.72802734375* proc.memory.percent‚17.823433876037598 disk.sda3.in‚ 0.8359375( disk.nvme0c0n1.out‚65782.28564453125# disk.rbd1.out‚17772.34130859375 disk.rbd0.in‚0 network.sent‚ 8.566066e+06* disk./.usagePercent‚39.200000110595255$ disk./.usageGB‚584.1083297729492 disk.sda1.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚65782.28564453125) proc.memory.availableMB‚ 1883.86328125 cpu‚25.586980349084627 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚15598.80078125 disk.sda2.in‚0¸°g2ˆ- _wandbruntime‚7562  _runtime‚7562ÊÞóûɉà _step‚371 ' lr-AdamW/pg1‚0.00024486491510129825 ' lr-AdamW/pg2‚0.00024486491510129825 trainer/global_step‚4549 % _timestamp‚1.7843273089687443e+09 _runtime‚7562.343399955‰ÈBFÎŠÈ ' lr-AdamW/pg1‚0.00024486491510129825 ' lr-AdamW/pg2‚0.00024486491510129825 trainer/global_step‚4549 % _timestamp‚1.7843273089687443e+09 _runtime‚7562.343399955 _step‚371óÌÇž-2‹- _wandbruntime‚7562  _runtime‚7562RÖà5þ'ŒøO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1311378278975488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1307098593861632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38793731 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38793737 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353284141019136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192254 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39398517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12924484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358805091915776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11672952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4278313550848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1307098620035072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4279658940416 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1307098620035072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311376907412480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11672947 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310804569881613 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4272337118479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚4549 _runtime‚ 7562.34459424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24597436 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358804687191040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78182275 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4279658940416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5520546171904 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5520582805504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4270992518157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚372 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278313550848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311378278975488è=LÚ(ýO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5520582805504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4270992518157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚372 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278313550848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311378278975488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1311378278975488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1307098593861632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38793731 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38793737 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353284141019136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192254 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39398517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12924484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358805091915776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11672952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4278313550848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1307098620035072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4279658940416 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1307098620035072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311376907412480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11672947 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310804569881613 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4272337118479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚4549 _runtime‚ 7562.34459424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24597436 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358804687191040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78182275 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4279658940416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5520546171904ô€UUà2Ž- _wandbruntime‚7563  _runtime‚75634Ð,ó‚N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38801943 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311084745154437 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1311657204748288 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311655833178112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38801937 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4280583983104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198959 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1307376594591744 TEDeviceStatsMonitor.on_train_batch_end/alloca¸ó'ted_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24602422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353361995517952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39407009 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358883765109760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5521769591808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353361627426816 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198959 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11675490 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4273260584837 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38801937 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1307376594591744 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12926932 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5521732965376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1311083375126035 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38801943 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚373 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1307376620765184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4279238586368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4279238586368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1306811484569600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚4549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚7563.727732551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4271915984403 $ _timestamp‚1.784327310352793e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39397022 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640Ž)-u '‡N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1306811484569600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚4549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚7563.727732551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4271915984403 $ _timestamp‚1.784327310352793e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39397022 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38801943 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311084745154437 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1311657204748288 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311655833178112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38801937 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4280583983104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198959 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1307376594591744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24602422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353361995517952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39407009 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358883765109760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5521769591808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353361627426816 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198959 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11675490 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4273260584837 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38801937 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1307376594591744 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12926932 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5521732965376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1311083375126035 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38801943 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚373 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1307376620765184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4279238586368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4279238586368õ%gÃ2‘- _wandbruntime‚7563  _runtime‚75630<…µ ’„ 0loss/FRt2/assignment_loss‚1.7639368296943903 trainer/global_step‚4549 % _timestamp‚1.7843273104387796e+09 0loss/FRt1/assignment_loss‚0.8719590998244011 2loss/SRqqt1/assignment_loss‚0.5418408999765215 /loss/FRt1/detection_loss‚1.5118036134436128 /loss/FBt1/detection_loss‚0.4921393698947858 /loss/FBt2/detection_loss‚0.8899392683345538 2loss/SRqqt2/assignment_loss‚0.5342093589456742 2loss/FBt1/assignment_loss‚0.013855829478798065 1loss/SRqqt1/detection_loss‚0.6240629520922385 _runtime‚ 7563.81311595 _step‚374 3loss/FBt2/assignment_loss‚0.0026647222399560633 /loss/FRt2/detection_loss‚0.5733246850636299 1loss/SRqqt2/detection_loss‚0.9814389547158693 %loss/total_loss‚8.801175583704431 epoch‚11QQM®“‰ 0loss/FRt1/assignment_loss‚0.8719590998244011 2loss/SRqqt1/assignment_loss‚0.5418408999765215 /loss/FRt1/detection_loss‚1.5118036134436128 /loss/FBt1/detection_loss‚0.4921393698947858 /loss/FBt2/detection_loss‚0.8899392683345538 2loss/SRqqt2/assignment_loss‚0.5342093589456742 2loss/FBt1/assignment_loss‚0.013855829478798065 1loss/SRqqt1/detection_loss‚0.6240629520922385 _runtime‚ 7563.81311595 _step‚374 3loss/FBt2/assignment_loss‚0.0026647222399560633 /loss/FRt2/detection_loss‚0.5733246850636299 1loss/SRqqt2/detection_loss‚0.9814389547158693 %loss/total_loss‚8.801175583704431 epoch‚11 0loss/FRt2/assignment_loss‚1.7639368296943903 trainer/global_step‚4549 % _timestamp‚1.7843273104387796e+09öûhk«””‚(:‰ ™ÙêÒòõž¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚003¶:•‚(:¯ ™ÙêÒ·Žç® disk.rbd0.in‚0 disk.rbd2.out‚54.72802734375 cpu‚25.58729984040397$ disk./.usageGB‚584.1100234985352 disk.sda1.out‚0 disk.sda.out‚ 443.29296875 disk.sda2.in‚0# proc.memory.rssMB‚ 2920.19140625 network.sent‚ 8.571126e+06& disk.nvme0n1.out‚65789.86279296875( proc.memory.availableMB‚ 1883.6328125* proc.memory.percent‚17.823433876037598 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 443.29296875# disk.nvme0c0n1.in‚ 15599.6328125 disk.sda.in‚ 0.8359375 network.recv‚ 1.95116e+06 disk.rbd2.in‚0.015625# disk.dm-0.out‚65789.86279296875 disk.sda3.out‚ 443.29296875 disk.rbd1.in‚ 506.52734375 disk.sda3.in‚ 0.8359375 disk.sda1.in‚0# disk.rbd1.out‚17772.48193359375 disk.rbd0.out‚0! disk.nvme0n1.in‚ 15599.6328125) disk./.usagePercent‚39.20011377794046 disk.dm-0.in‚ 15599.6328125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚65789.86279296875 disk.sda2.out‚0$ memory_percent‚88.50321769714355̈«ž–‚(:“ ¨ÙêÒà‰“¬ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚301.872' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.37440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þð?>—‚(:³ ¨ÙêÒÐêñ®* proc.memory.percent‚17.823433876037598 network.sent‚ 8.633743e+06 disk.rbd0.in‚0 disk.rbd2.out‚54.72802734375 disk.sda1.out‚0 disk.sda3.out‚ 443.51171875 disk.rbd1.in‚ 506.52734375 cpu‚25.58745577172369 disk.dm-1.in‚ 0.8359375 disk.sda2.out‚0# proc.memory.rssMB‚ 2920.19140625 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚65800.43310546875# memory_percent‚88.5028600692749) proc.memory.availableMB‚ 1883.69140625) disk./.usagePercent‚39.20011992212128 disk.rbd0.out‚0 disk.sda3.in‚ 0.8359375 disk.sda.out‚ 443.51171875 disk.sda2.in‚0 network.recv‚ 1.956183e+06$ disk./.usageGB‚584.1101150512695 disk.sda.in‚ 0.8359375 disk.drbd2165.out‚0 disk.dm-1.out‚ 443.51171875& disk.nvme0n1.out‚65800.43310546875# disk.dm-0.out‚65800.43310546875# disk.rbd1.out‚17772.48193359375 disk.dm-0.in‚15738.15234375$ disk.nvme0c0n1.in‚15738.15234375" disk.nvme0n1.in‚15738.15234375Ó­±”˜‚(:‰ ·ÙêÒƒ“‹¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚303.759' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.7518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×5;µ0™‚(:¥ ·ÙêÒÂñÚ® disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 8.63891e+06 disk.rbd2.out‚54.72802734375 disk.drbd2165.out‚0$ proc.memory.availableMB‚1883.625# proc.memory.rssMB‚ 2920.19140625 disk.sda2.out‚0$ disk./.usageGB‚584.1099014282227 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625$ disk.nvme0n1.out‚65805.037109375 disk.sda.out‚ 443.7734375) disk./.usagePercent‚39.20010558569936 disk.sda3.out‚ 443.7734375 disk.sda.in‚ 0.8359375 disk.dm-0.in‚15738.15234375 disk.sda2.in‚0# disk.rbd1.out‚17772.54443359375 disk.dm-1.out‚ 443.7734375$ disk.nvme0c0n1.in‚15738.15234375& disk.nvme0c0n1.out‚65805.037109375" disk.nvme0n1.in‚15738.15234375 disk.rbd0.in‚0! disk.dm-0.out‚65805.037109375 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375* proc.memory.percent‚17.823433876037598$ memory_percent‚88.50326538085938 cpu‚25.587579823899606 network.recv‚ 1.95981e+06 disk.sda1.in‚0“ái±6š‚(:« ÆÙêÒê컯* proc.memory.percent‚17.823433876037598 cpu‚25.587893418766335 network.recv‚ 1.962748e+06 disk.sda1.out‚0 disk.sda3.out‚ 444.00390625' disk.nvme0c0n1.out‚65816.1123046875% disk.nvme0n1.out‚65816.1123046875 proc.cpu.threads‚17) disk./.usagePercent‚39.20011505797813 disk.rbd0.in‚0 disk.dm-0.in‚ 15738.984375 disk.sda2.in‚0$ disk./.usageGB‚584.1100425720215" disk.nvme0c0n1.in‚ 15738.984375 disk.sda2.out‚0) proc.memory.availableMB‚ 1883.55078125 disk.rbd2.in‚0.015625" disk.dm-0.out‚65816.1123046875 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 444.00390625 disk.rbd1.in‚ 506.52734375 network.sent‚ 8.64399e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.8359375# proc.memory.rssMB‚ 2920.19140625# disk.rbd1.out‚17772.54443359375 disk.rbd2.out‚54.72802734375$ memory_percent‚88.50371837615967 disk.nvme0n1.in‚ 15738.984375 disk.sda.in‚ 0.8359375 disk.sda.out‚ 444.003906257¨'p”›‚(:‰ ÆÙêÒ€¨¯¯ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò„~OŸœ‚(:” ÕÙêÒÁü¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚299.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.948800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž½óy)‚(:ž ÕÙêÒ…¦ê® disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.dm-1.in‚ 0.8359375& disk.nvme0n1.out‚65819.71240234375 disk.sda2.in‚0# disk.rbd1.out‚17772.66943359375 cpu‚25.588117198862466 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625# disk.dm-0.out‚65819.71240234375 disk.dm-1.out‚ 444.30859375$ memory_percent‚88.50469589233398) disk./.usagePercent‚39.20013707462607 disk.sda.in‚ 0.8359375 disk.sda.out‚ 444.30859375 disk.sda1.out‚0 disk.sda3.out‚ 444.30859375' proc.memory.availableMB‚ 1883.390625 network.recv‚ 1.966733e+06$ disk./.usageGB‚584.1103706359863 disk.rbd2.out‚55.53662109375 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2920.19140625* proc.memory.percent‚17.823433876037598 disk.nvme0c0n1.in‚15739.5( disk.nvme0c0n1.out‚65819.71240234375 disk.sda2.out‚0 disk.dm-0.in‚15739.5 disk.sda3.in‚ 0.8359375 proc.cpu.threads‚17 network.sent‚ 8.649088e+06 disk.nvme0n1.in‚15739.5 disk.drbd2165.in‚0»îæ±2ž- _wandbruntime‚7641  _runtime‚76411ã•ÇŸÁ ' lr-AdamW/pg1‚0.00024418377170460736 ' lr-AdamW/pg2‚0.00024418377170460736 trainer/global_step‚4599 % _timestamp‚1.7843273882537105e+09 _runtime‚ 7641.6283281 _step‚375Ü{ÿÌ Æ ' lr-AdamW/pg1‚0.00024418377170460736 ' lr-AdamW/pg2‚0.00024418377170460736 trainer/global_step‚4599 % _timestamp‚1.7843273882537105e+09 _runtime‚ 7641.6283281 _step‚375÷˜ Úx2¡- _wandbruntime‚7641  _runtime‚7641Óþ9`¢ùO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39204037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4324629481472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843273882547498e+09 _runtime‚7641.629516113 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5579983065088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1325323259848704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24861732 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11800102 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4318574231587 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4317229631265 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4324629481472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/r›UKïeserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39814296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13061902 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357158829494272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320998630367232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39824269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39204037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362738849192960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4325974871040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24862004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1325324631411712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4325974871040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362738444468224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357158461403136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1320998656540672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028306 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1324746265931811 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325323259848704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13061635 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320998656540672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320998630367232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39204031 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0~¸ìh(£þO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39824269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1324744895903521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39204031 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1320427691700224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1320427666272256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1325323259848704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24861732 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11800102 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4318574231587 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4317229631265 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4324629481472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39814296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13061902 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357158829494272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320998630367232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39824269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39204037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362738849192960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4325974871040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24862004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1325324631411712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4325974871040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362738444468224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357158461403136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1320998656540672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028306 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1324746265931811 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325323259848704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13061635 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320998656540672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320998630367232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39204031 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39204037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4324629481472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843273882547498e+09 _runtime‚7641.629516113 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5579983065088 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13064402 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5581203756032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045004 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4318153097511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321276631097344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045004 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362817114940416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357236315901952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357235947810816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1320705573478400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39822774 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843273896487277e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24867042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39212243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11802640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321276631097344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39212243 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39832761 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321276657270784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39832761 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035011 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24867042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39212243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11802640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321276631097344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39212243 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39832761 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321276657270784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39832761 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035011 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13064402 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5581203756032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045004 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4318153097511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321276631097344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045004 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362817114940416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357236315901952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357235947810816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1320705573478400ùÐÙ(É2§- _wandbruntime‚7643  _runtime‚7643J;; ¨ƒ 0loss/FRt2/assignment_loss‚1.6810186726821434 2loss/SRqqt1/assignment_loss‚0.5181670892785517 2loss/SRqqt2/assignment_loss‚0.5783442166290762 1loss/SRqqt1/detection_loss‚0.6637554687789965 1loss/SRqqt2/detection_loss‚0.8677646418386483 /loss/FBt2/detection_loss‚0.7780655211369906 % _timestamp‚1.7843273897310529e+09 3loss/FBt2/assignment_loss‚0.0007250680999817405 /loss/FRt2/detection_loss‚0.6080077406640926 .loss/FBt1/detection_loss‚0.548488635380097 trainer/global_step‚4599 _runtime‚7643.105586051 _step‚378 /loss/FRt1/assignment_loss‚0.770241235960842 2loss/FBt1/assignment_loss‚0.014182314802807147 /loss/FRt1/detection_loss‚1.6475635665649901 %loss/total_loss‚8.676324171817217 epoch‚11vÆ*i©ˆ /loss/FRt2/detection_loss‚0.6080077406640926 .loss/FBt1/detection_loss‚0.548488635380097 trainer/global_step‚4599 _runtime‚7643.105586051 _step‚378 /loss/FRt1/assignment_loss‚0.770241235960842 2loss/FBt1/assignment_loss‚0.014182314802807147 /loss/FRt1/detection_loss‚1.6475635665649901 %loss/total_loss‚8.676324171817217 epoch‚11 0loss/FRt2/assignment_loss‚1.6810186726821434 2loss/SRqqt1/assignment_loss‚0.5181670892785517 2loss/SRqqt2/assignment_loss‚0.5783442166290762 1loss/SRqqt1/detection_loss‚0.6637554687789965 1loss/SRqqt2/detection_loss‚0.8677646418386483 /loss/FBt2/detection_loss‚0.7780655211369906 % _timestamp‚1.7843273897310529e+09 3loss/FBt2/assignment_loss‚0.0007250680999817405ú…@Ê蔪‚(:‰ äÙêÒ–ù ­ gpu.0.gpu‚38.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áø9}@«‚(:µ äÙêÒ’ο® cpu‚25.588468476643886 disk.sda3.out‚ 444.52734375 disk.sda2.in‚0 disk.dm-1.in‚ 0.8359375 proc.cpu.threads‚17 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚15812.15234375 disk.sda.out‚ 444.52734375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0( disk.nvme0c0n1.out‚65828.40869140625" disk.nvme0n1.in‚15812.15234375# disk.rbd1.out‚17772.66943359375* disk./.usagePercent‚39.200157811236345) proc.memory.availableMB‚ 1883.54296875# proc.memory.rssMB‚ 2920.19140625 disk.sda3.in‚ 0.8359375 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 444.52734375$ memory_percent‚88.50376605987549# disk.dm-0.out‚65828.40869140625 network.sent‚ 8.65415e+06$ disk./.usageGB‚584.1106796264648& disk.nvme0n1.out‚65828.40869140625 disk.sda.in‚ 0.8359375 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.823433876037598 network.recv‚ 1.970309e+06 disk.rbd2.out‚55.54052734375 disk.dm-0.in‚15812.15234375ž@^”¬‚(:‰ óÙêÒ¬†¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"cØe6­‚(:« óÙêÒ ÕŠ® disk.sda2.out‚0 disk.rbd2.out‚ 55.6591796875) disk./.usagePercent‚39.20015422713087 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 cpu‚25.588721548140303( disk.nvme0c0n1.out‚65836.30908203125 disk.sda.out‚ 445.1484375 disk.rbd0.out‚0 disk.dm-0.in‚ 15812.984375 disk.dm-1.in‚ 0.8359375 disk.sda3.in‚ 0.8359375) proc.memory.availableMB‚ 1883.44921875# proc.memory.rssMB‚ 2920.19140625 disk.drbd2165.out‚0 disk.sda3.out‚ 445.1484375* proc.memory.percent‚17.823433876037598 proc.cpu.threads‚17 network.recv‚ 1.975751e+06" disk.nvme0c0n1.in‚ 15812.984375& disk.nvme0n1.out‚65836.30908203125# disk.rbd1.out‚17867.43505859375 disk.sda.in‚ 0.8359375 disk.sda1.in‚0 disk.nvme0n1.in‚ 15812.984375 disk.rbd0.in‚0# disk.dm-0.out‚65836.30908203125 disk.drbd2165.in‚0 network.sent‚ 8.717438e+06$ disk./.usageGB‚584.1106262207031 disk.rbd1.in‚ 506.52734375 disk.dm-1.out‚ 445.1484375$ memory_percent‚88.50433826446533,%?x’®‚(:‡ ‚ÚêÒºÔƬ gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.497' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¿V3¯‚(:¨ ‚ÚêÒýù°' disk.nvme0c0n1.out‚65841.4580078125 disk.sda.out‚ 445.4609375 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 cpu‚25.589118710769505 disk.sda3.in‚ 0.8359375 disk.nvme0n1.in‚ 15812.984375 disk.sda.in‚ 0.8359375# proc.memory.rssMB‚ 2920.19140625 proc.cpu.threads‚17 disk.sda3.out‚ 445.4609375 disk.rbd0.in‚0 disk.dm-0.in‚ 15812.984375 disk.drbd2165.out‚0 network.recv‚ 1.978954e+06% disk.nvme0n1.out‚65841.4580078125 disk.sda1.out‚0 disk.dm-1.in‚ 0.8359375$ memory_percent‚88.50395679473877* proc.memory.percent‚17.823433876037598" disk.nvme0c0n1.in‚ 15812.984375 disk.sda2.out‚0" disk.dm-0.out‚65841.4580078125 disk.rbd2.out‚ 55.6591796875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1883.51171875$ disk./.usageGB‚584.1105461120605# disk.rbd1.out‚17870.65380859375 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 445.4609375 network.sent‚ 8.723081e+06) disk./.usagePercent‚39.20014885097265 disk.rbd0.out‚0 disk.sda1.in‚0;ô¡ò“°‚(:ˆ ‘ÚêÒìÝ­¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ 7*±‚(:Ÿ ‘ÚêÒýŒú® disk.rbd0.out‚0 disk.rbd2.out‚ 55.7265625 disk.drbd2165.out‚0 disk.sda.in‚ 0.8359375 disk.sda.out‚445.625 disk.sda2.out‚0# disk.rbd1.out‚17998.79443359375 cpu‚25.589244966594954) disk./.usagePercent‚39.20014142675415# disk.dm-0.out‚65849.41162109375) proc.memory.percent‚17.86789894104004( disk.nvme0c0n1.out‚65849.41162109375" disk.nvme0n1.in‚15813.94140625& disk.nvme0n1.out‚65849.41162109375 disk.rbd1.in‚ 506.52734375$ memory_percent‚88.36827278137207$ disk./.usageGB‚584.1104354858398 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚445.625( proc.memory.availableMB‚ 1905.7421875 disk.dm-0.in‚15813.94140625 disk.dm-1.in‚ 0.8359375 disk.sda1.out‚0 disk.sda3.in‚ 0.8359375$ disk.nvme0c0n1.in‚15813.94140625 disk.sda2.in‚0" proc.memory.rssMB‚ 2927.4765625 proc.cpu.threads‚17 network.sent‚ 8.727613e+06 network.recv‚ 1.982235e+06 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.sda3.out‚445.625ØÞFΞ²‚(:“  ÚêÒ¸Èö« gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øb_ò.³‚(:£  ÚêҌ˺­ disk.sda2.in‚0 disk.rbd2.out‚ 55.7265625# disk.nvme0c0n1.in‚ 15910.2890625% disk.nvme0c0n1.out‚65860.87109375 disk.sda.in‚ 0.8359375 disk.dm-0.in‚ 15910.2890625 disk.dm-1.in‚ 0.8359375* proc.memory.percent‚17.918920516967773 cpu‚25.588920662732892 network.sent‚ 8.731753e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚65860.87109375* disk./.usagePercent‚39.200149106980184! disk.nvme0n1.in‚ 15910.2890625 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2935.8359375 proc.cpu.threads‚17 disk.dm-1.out‚ 446.01171875 disk.sda3.in‚ 0.8359375# disk.nvme0n1.out‚65860.87109375 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚88.33413124084473 disk.sda.out‚ 446.01171875# disk.rbd1.out‚18037.51318359375( proc.memory.availableMB‚ 1911.3359375 network.recv‚ 1.98468e+06$ disk./.usageGB‚584.1105499267578 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.sda3.out‚ 446.01171875Â#›2´- _wandbruntime‚7720  _runtime‚7720t ƒÂɵà trainer/global_step‚4649 % _timestamp‚1.7843274674268866e+09 _runtime‚7720.801542362 _step‚379 ' lr-AdamW/pg1‚0.00024349574622723266 ' lr-AdamW/pg2‚0.000243495746227232666Å©òÎ¶È trainer/global_step‚4649 % _timestamp‚1.7843274674268866e+09 _runtime‚7720.801542362 _step‚379 ' lr-AdamW/pg1‚0.00024349574622723266 ' lr-AdamW/pg2‚0.00024349574622723266ûjµºÑ2·- _wandbruntime‚7720  _runtime‚7720æú73ÿ'¸ùO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5639416976384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39614331 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39614337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11927247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4363466744373 trainer/global_step‚4649 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1339269612284928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864358 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1334898693046272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1339270983847936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5639453609984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1334321780609024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366672198763520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4364811344695 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1334898666872832 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40250021 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1339270983847936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1334321755181056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361032781787136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4370945412096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4372290801664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 _step‚380 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338685221925429 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39614331 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13199763 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338686591953719 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4372290801664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40240048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25127015 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 Ú\^è'¹þO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40240048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25127015 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5639416976384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39614331 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39614337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11927247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4363466744373 trainer/global_step‚4649 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1339269612284928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864358 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1334898693046272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1339270983847936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5639453609984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1334321780609024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366672198763520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4364811344695 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1334898666872832 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40250021 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1339270983847936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1334321755181056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361032781787136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4370945412096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4372290801664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 _step‚380 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338685221925429 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39614331 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13199763 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338686591953719 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4372290801664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment±$϶.large_pool.peak‚1175üHV6‹2º- _wandbruntime‚7722  _runtime‚7722ß—²'»‚N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39622543 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39622537 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4371870447616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1335176667602944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25132294 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13202499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25132035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11929795 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338964027169851 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1334599636959232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871063 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4371870447616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39622543 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881056 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40248526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5640609281024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1335176693776384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40258513 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11929790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4365734811053 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7722.196986329 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5640645907456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1334599662387200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4649 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361110636285952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40248526 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4373215844352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1335176667602944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4373215844352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1335176693776384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402@Îç@ '¼‡N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366751282193408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40248526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5640609281024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1335176693776384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40258513 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11929790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4365734811053 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7722.196986329 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5640645907456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1334599662387200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4649 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361110636285952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40248526 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4373215844352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1335176667602944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4373215844352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1335176693776384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39622543 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39622537 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4371870447616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1335176667602944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25132294 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13202499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25132035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11929795 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338964027169851 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1334599636959232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871063 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4371870447616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39622543 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881056 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40024631 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364907470262272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4409703857481 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40675773 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40024631 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40665800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 trainer/global_step‚4699 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80700410 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonit<ÖAor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1353215964721152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843275461183739e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40665800 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353215964721152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370605912103936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5698809932800 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1348215869517824 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40024631 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364907470262272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4409703857481 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40675773 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40024631 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40665800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 trainer/global_step‚4699 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80700410 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1353215964721152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843275461183739e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40665800 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353215964721152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370605912103936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5698809932800 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1348215869517824 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5699990959104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364984588578816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40032837 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4419531774976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40674278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1348493725868032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1349076704108544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40684265 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25396776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4411971924161 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4418186378240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1352905723220161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40684265 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370684984255488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364984956669952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1352904353191759 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚385 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12056940 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717108 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1353496262056960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40032843 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4419531774976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13339836 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40032837 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12056945 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1349076704108544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843275475055094e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1348493751296000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 trainer/global_step‚4699 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4410627323727 _runtime‚7800.880372537 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40032843 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1349076730281984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4418186378240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370684579537920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25397037 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13340092 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40032837 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4419531774976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40674278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1348493725868032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1349076704108544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40684265 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25396776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4411971924161 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4418186378240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1352905723220161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40684265 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370684984255488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364984956669952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1352904353191759 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚385 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.×0èùcurrent‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12056940 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717108 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1353496262056960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40032843 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4419531774976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13339836 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40032837 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12056945 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1349076704108544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843275475055094e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1348493751296000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 trainer/global_step‚4699 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4410627323727 _runtime‚7800.880372537 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40032843 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1349076730281984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4418186378240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370684579537920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25397037 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13340092 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5699990959104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364984588578816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FÖ‚ 2loss/FBt2/assignment_loss‚0.012520816765710189 trainer/global_step‚4699 /loss/FBt1/detection_loss‚0.5650392072430501 %loss/total_loss‚9.010188282782671 _runtime‚7800.963668248 .loss/FRt1/detection_loss‚1.499267240926962 1loss/SRqqt1/detection_loss‚0.6009294918069162 1loss/SRqqt2/detection_loss‚0.9576054252250167 /loss/FBt2/detection_loss‚1.1294501878049534 epoch‚11 _step‚386 0loss/FRt2/assignment_loss‚1.7507806805739725 /loss/FRt2/detection_loss‚0.6070314769540517 % _timestamp‚1.7843275475893056e+09 0loss/FRt1/assignment_loss‚0.7432310838590969 2loss/SRqqt1/assignment_loss‚0.5447614846730557 2loss/SRqqt2/assignment_loss‚0.5673435040558504 1loss/FBt1/assignment_loss‚0.0322276828940346252” ׇ %loss/total_loss‚9.010188282782671 _runtime‚7800.963668248 .loss/FRt1/detection_loss‚1.499267240926962 1loss/SRqqt1/detection_loss‚0.6009294918069162 1loss/SRqqt2/detection_loss‚0.9576054252250167 /loss/FBt2/detection_loss‚1.1294501878049534 epoch‚11 _step‚386 0loss/FRt2/assignment_loss‚1.7507806805739725 /loss/FRt2/detection_loss‚0.6070314769540517 % _timestamp‚1.7843275475893056e+09 0loss/FRt1/assignment_loss‚0.7432310838590969 2loss/SRqqt1/assignment_loss‚0.5447614846730557 2loss/SRqqt2/assignment_loss‚0.5673435040558504 1loss/FBt1/assignment_loss‚0.03222768289403462 2loss/FBt2/assignment_loss‚0.012520816765710189 trainer/global_step‚4699 /loss/FBt1/detection_loss‚0.5650392072430501‚Ç|Õ¬”Ø‚(:‰ ‰ÛêÒ­¡¥¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç*Å'Ù‚(:œ ‰ÛêÒ¤ˆå®$ memory_percent‚88.33901882171631 cpu‚25.58609431799045 disk.sda.in‚ 0.8359375# disk.rbd1.out‚18454.06005859375 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2936.69140625 network.recv‚ 2.012703e+06( disk.nvme0c0n1.out‚65917.23486328125) disk./.usagePercent‚39.20023589353429) proc.memory.availableMB‚ 1910.53515625* proc.memory.percent‚17.924141883850098 disk.dm-1.in‚ 0.8359375 disk.rbd2.out‚ 58.72265625 network.sent‚ 8.873521e+06 disk.nvme0c0n1.in‚16078.25 disk.sda.out‚ 448.0859375 disk.rbd0.out‚0& disk.nvme0n1.out‚65917.23486328125 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.dm-0.in‚16078.25 proc.cpu.threads‚17$ disk./.usageGB‚584.1118431091309 disk.dm-1.out‚ 448.0859375 disk.sda3.out‚ 448.0859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375 disk.nvme0n1.in‚16078.25 disk.sda2.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚65917.23486328125 disk.sda1.out‚0Y±Ú‚(:’ ˜ÛêÒ­Þ´¬ gpu.0.gpu‚28.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚345.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.17559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+{©è*Û‚(:Ÿ ˜ÛêÒ´À¯ disk.rbd0.in‚0 disk.dm-0.out‚65928.59765625# disk.nvme0n1.out‚65928.59765625 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.rbd2.out‚ 58.72265625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.8359375 disk.sda3.out‚ 448.21875% disk.nvme0c0n1.out‚65928.59765625* proc.memory.percent‚17.924141883850098 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚16079.20703125 network.sent‚ 8.925445e+06) disk./.usagePercent‚39.20023128539867$ disk./.usageGB‚584.1117744445801 disk.sda2.in‚0# proc.memory.rssMB‚ 2936.69140625 network.recv‚ 2.017216e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375" disk.nvme0n1.in‚16079.20703125 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.dm-1.out‚ 448.21875 disk.sda.in‚ 0.8359375 disk.sda.out‚ 448.21875 cpu‚25.585800338331534 disk.dm-0.in‚16079.20703125# disk.rbd1.out‚18688.23193359375# memory_percent‚88.3411169052124) proc.memory.availableMB‚ 1910.19140625jGŠØ”Ü‚(:‰ §ÛêÒ¨´­ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J6¡2Ý‚(:§ §ÛêÒ¨úø°# disk.dm-0.out‚65931.71728515625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16283.609375& disk.nvme0n1.out‚65931.71728515625 disk.sda3.in‚ 0.8359375 disk.sda.in‚ 0.8359375 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625( disk.nvme0c0n1.out‚65931.71728515625* proc.memory.percent‚17.924141883850098 proc.cpu.threads‚17 network.sent‚ 8.930534e+06 disk.drbd2165.out‚0$ memory_percent‚88.34190368652344 network.recv‚ 2.021066e+06 disk.sda.out‚ 448.4296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375# proc.memory.rssMB‚ 2936.69140625) disk./.usagePercent‚39.20022155711237 disk.sda2.in‚0# disk.rbd1.out‚18746.77880859375 disk.dm-1.out‚ 448.4296875 disk.sda3.out‚ 448.4296875" disk.nvme0c0n1.in‚ 16283.609375 disk.rbd0.out‚0% proc.memory.availableMB‚ 1910.0625 cpu‚25.585402180029686# disk./.usageGB‚584.111629486084 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 16283.609375ÕzI”Þ‚(:‰ ¶ÛêÒ”À¤¬ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v8-‰;ß‚(:° ¶ÛêҔˮ disk.rbd2.in‚0.015625 disk.dm-0.in‚16284.44140625 disk.sda1.out‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 448.76953125" disk.nvme0n1.in‚16284.44140625 network.sent‚ 8.935589e+06 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375* proc.memory.percent‚17.924141883850098 proc.cpu.threads‚17 network.recv‚ 2.024454e+06" disk.dm-0.out‚65941.4814453125% disk.nvme0n1.out‚65941.4814453125 disk.sda.out‚ 448.76953125 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125( proc.memory.availableMB‚ 1909.6796875 cpu‚25.58506863579466$ disk./.usageGB‚584.1090126037598' disk.nvme0c0n1.out‚65941.4814453125$ memory_percent‚88.34424018859863 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.8359375 disk.sda3.out‚ 448.76953125$ disk.nvme0c0n1.in‚16284.44140625 disk.drbd2165.out‚0) disk./.usagePercent‚39.20004593594388 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2936.69140625 disk.sda.in‚ 0.8359375 disk.sda2.out‚0# disk.rbd1.out‚18805.48193359375 S°6”à‚(:‰ ÅÛêÒ¦îë« gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â=n.á‚(:£ ÅÛêÒ‡äË®$ disk./.usageGB‚584.1089019775391 disk.rbd0.in‚0! disk.dm-0.out‚65949.189453125 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.8359375 disk.dm-1.out‚ 449.03125 disk.sda.out‚ 449.03125 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2936.69140625 network.recv‚ 2.02793e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.8359375* proc.memory.percent‚17.924141883850098! disk.nvme0n1.in‚ 16285.3984375 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 16285.3984375 disk.sda1.in‚0 disk.sda.in‚ 0.8359375# disk.rbd1.out‚18819.09130859375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625# disk.nvme0c0n1.in‚ 16285.3984375& disk.nvme0c0n1.out‚65949.189453125 disk.sda2.in‚0$ memory_percent‚88.34435939788818 network.sent‚ 8.940685e+06* disk./.usagePercent‚39.200038511725396 disk.sda3.out‚ 449.03125$ disk.nvme0n1.out‚65949.189453125 disk.rbd1.in‚ 506.52734375) proc.memory.availableMB‚ 1909.66015625 cpu‚25.584682261680417 proc.cpu.threads‚17ñò0?2â- _wandbruntime‚7879  _runtime‚7879"¶j8Éãà ' lr-AdamW/pg1‚0.00024209922623197975 ' lr-AdamW/pg2‚0.00024209922623197975 trainer/global_step‚4749 % _timestamp‚1.7843276256401737e+09 _runtime‚7879.014720647 _step‚387Šó ÎäÈ _runtime‚7879.014720647 _step‚387 ' lr-AdamW/pg1‚0.00024209922623197975 ' lr-AdamW/pg2‚0.00024209922623197975 trainer/global_step‚4749 % _timestamp‚1.7843276256401737e+09ƒ 3Z%2å- _wandbruntime‚7879  _runtime‚7879¿‰™|þ'æøO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368781422555136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4463577273344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1367162317157376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374540014264320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362698739884032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13475491 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362698766057472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1362109958426624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4457285570911 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5758626245632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4463577273344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368781790646272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536462 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40434931 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25657289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12181547 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1367163688720384 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40434931 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41091552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526483 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚4749 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1366567243997535 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚388 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1367163688720384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1362109932998656 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362698766057472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40434937 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40434937 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4464922662912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41091552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455940970589 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 7879.01597223 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41101525 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12181552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374540416891904 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1367162317157376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25657038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1362698739884032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973Sy"}ÚçýO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362698739884032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13475491 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362698766057472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1362109958426624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4457285570911 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5758626245632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4463577273344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368781790646272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536462 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40434931 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25657289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12181547 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1367163688720384 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40434931 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41091552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526483 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚4749 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1366567243997535 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚388 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1367163688720384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1362109932998656 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362698766057472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40434937 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40434937 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4464922662912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41091552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455940970589 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 7879.01597223 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41101525 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12181552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374540416891904 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1367162317157376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25657038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1362698739884032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368781422555136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4463577273344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1367162317157376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374540014264320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995„9û2è- _wandbruntime‚7880  _runtime‚7880K¬Øš 'éƒN REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40443143 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368858908962816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4458209037269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 trainer/global_step‚4749 _runtime‚7880.410959668 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1367441242923008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1366844679213667 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1362387840204800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚389 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1366846049242069 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41100030 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12184095 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543167 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41110017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374619106199552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368859277053952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843276270360162e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40443143 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40443137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25662282 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4464502308864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13478192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40443137 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25662523 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1362976766787584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1362976766787584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456864436835l2e\'êˆN C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843276270360162e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40443143 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40443137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25662282 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4464502308864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13478192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40443137 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25662523 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1362976766787584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1362976766787584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456864436835 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40443143 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368858908962816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4458209037269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 trainer/global_step‚4749 _runtime‚7880.410959668 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1367441242923008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1366844679213667 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1362387840204800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚389 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1366846049242069 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41100030 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12184095 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543167 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41110017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374619106199552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368859277053952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40845237 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1380506199991153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4511238593536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4511238593536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41527277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372656111030272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1380507570019443 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381108669593600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40845237 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1376598776389632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5818962766848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7959.163711172 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40845231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13613656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362535 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13613401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40845231 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25922358 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4503522684019 trainer/global_step‚4799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82372514 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376004021907456 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1381108669593600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41517304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12308697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5818928230400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4502178083697 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1381110041156608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378474671169536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372655742939136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1376004047335424 % _timestamp‚1.7843277057889135e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509893203968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41517304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ê÷(ýþO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12308697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5818928230400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4502178083697 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1381110041156608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378474671169536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372655742939136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1376004047335424 % _timestamp‚1.7843277057889135e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509893203968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41517304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1376598802563072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25922098 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1381110041156608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1376598802563072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1376598776389632 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40845237 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1380506199991153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4511238593536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4511238593536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41527277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372656111030272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1380507570019443 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381108669593600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40845237 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1376598776389632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5818962766848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7959.163711172 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40845231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13613656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362535 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13613401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40845231 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25922358 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4503522684019 trainer/global_step‚4799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82372514 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376004021907456 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1381108669593600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41517304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008ˆeÐG2þ- _wandbruntime‚7960  _runtime‚7960/“8Þ 'ÿƒN HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1380786375263977 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376281929113600 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1376876777119744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372733597437952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4504446150377 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1376876803293184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379219 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25927392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13616152 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25927641 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5820156262400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1380785005235575 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389212 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1376281903685632 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1376876803293184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40853443 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4503101549943 _runtime‚7960.562873592 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41535769 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12311240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40853437 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510818239488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41525782 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525782 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12311245 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4512163636224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÐMk¸€ˆN J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40853437 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510818239488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41525782 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525782 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12311245 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4512163636224 TFDeviceStatsMonitor.on_traá|¼û in_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1380786375263977 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376281929113600 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1376876777119744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372733597437952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4504446150377 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1376876803293184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379219 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25927392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13616152 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25927641 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5820156262400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1380785005235575 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389212 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1376281903685632 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1376876803293184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40853443 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4503101549943 _runtime‚7960.562873592 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41535769 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12311240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‰œRÇ2- _wandbruntime‚7960  _runtime‚7960ô0š@ ‚ƒ /loss/FRt2/detection_loss‚0.5558817649020057 /loss/FBt2/detection_loss‚0.5167477176589674 %loss/total_loss‚8.280696502898508 _runtime‚7960.644070252 _step‚394 2loss/SRqqt1/assignment_loss‚0.5230307378771669 2loss/SRqqt2/assignment_loss‚0.5394924806216295 2loss/FBt1/assignment_loss‚0.012490557286669937 /loss/FRt1/assignment_loss‚0.727314433351678 2loss/FBt2/assignment_loss‚0.008271870828956031 /loss/FRt1/detection_loss‚1.5549754209316045 trainer/global_step‚4799 1loss/SRqqt1/detection_loss‚0.5931603377342073 0loss/SRqqt2/detection_loss‚1.005283285479527 0loss/FBt1/detection_loss‚0.36971581239961615 epoch‚11 % _timestamp‚1.7843277072695134e+09 0loss/FRt2/assignment_loss‚1.8743320838264796ßç.ƒˆ 2loss/SRqqt1/assignment_loss‚0.5230307378771669 2loss/SRqqt2/assignment_loss‚0.5394924806216295 2loss/FBt1/assignment_loss‚0.012490557286669937 /loss/FRt1/assignment_loss‚0.727314433351678 2loss/FBt2/assignment_loss‚0.008271870828956031 /loss/FRt1/detection_loss‚1.5549754209316045 trainer/global_step‚4799 1loss/SRqqt1/detection_loss‚0.5931603377342073 0loss/SRqqt2/detection_loss‚1.005283285479527 0loss/FBt1/detection_loss‚0.36971581239961615 epoch‚11 % _timestamp‚1.7843277072695134e+09 0loss/FRt2/assignment_loss‚1.8743320838264796 /loss/FRt2/detection_loss‚0.5558817649020057 /loss/FBt2/detection_loss‚0.5167477176589674 %loss/total_loss‚8.280696502898508 _runtime‚7960.644070252 _step‚394ŠÙO%SA„‚(:¶ ŸÜêÒçŠö® disk.drbd2165.out‚0$ memory_percent‚88.34443092346191) proc.memory.percent‚17.93220043182373 cpu‚25.583788731195625 disk.sda.out‚ 451.44921875 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125( proc.memory.availableMB‚ 1909.6484375 disk.dm-0.in‚16601.66015625 disk.sda1.in‚0 disk.dm-1.out‚ 451.44921875# proc.memory.rssMB‚ 2938.01171875 network.sent‚ 9.029506e+06( disk.nvme0c0n1.out‚65996.23291015625 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19240.74755859375$ disk./.usageGB‚584.1116714477539 disk.sda3.in‚ 5.93359375 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.050303e+06 disk.sda3.out‚ 451.44921875" disk.nvme0n1.in‚16601.66015625& disk.nvme0n1.out‚65996.23291015625 disk.sda.in‚ 5.93359375$ disk.nvme0c0n1.in‚16601.66015625 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375) disk./.usagePercent‚39.20022437319525 disk.rbd0.out‚0 disk.rbd2.in‚0.015625# disk.dm-0.out‚65996.23291015625;8å…‚(:… ŸÜêÒÀ¥Í° gpu.0.gpu‚4.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚330.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»±_Ò”†‚(:‰ ®ÜêÒ„§É¬ gpu.0.gpu‚78.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î//ÃC‡‚(:¸ ®ÜêÒ«Õ²® network.recv‚ 2.055464e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 451.82421875 disk.sda.out‚ 451.82421875 disk.sda2.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚584.1115951538086 disk.sda1.out‚0( disk.nvme0c0n1.out‚66001.56494140625# proc.memory.rssMB‚ 2939.33203125 disk.dm-0.in‚16601.66015625# disk.rbd1.out‚19342.20068359375 disk.drbd2165.in‚0 disk.dm-1.out‚ 451.82421875 disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚16601.66015625& disk.nvme0n1.out‚66001.56494140625 disk.sda.in‚ 5.93359375* proc.memory.percent‚17.940258979797363 cpu‚25.58844486069726 network.sent‚ 9.092614e+06 disk.rbd2.in‚0.015625# disk.dm-0.out‚66001.56494140625 disk.rbd0.in‚0) proc.memory.availableMB‚ 1908.37109375 disk.sda1.in‚0 disk.dm-1.in‚ 5.93359375" disk.nvme0n1.in‚16601.66015625 disk.rbd1.in‚ 506.52734375 disk.sda2.out‚0* disk./.usagePercent‚39.200219253044565 disk.rbd2.out‚58.78564453125$ memory_percent‚88.35222721099854 proc.cpu.threads‚17ÌÛ2ˆ- _wandbruntime‚7987  _runtime‚7987 G-Xò=‰ì{ 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9930832982063293 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6233391761779785 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.143853098154068 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18196415901184082 epoch‚11 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16920924186706543 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1604541540145874 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7411731481552124 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.621074378490448 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05945893004536629 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7389044761657715 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06187319755554199 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.14236336946487427 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28552260994911194 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5717385411262512 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3641650378704071 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12861086428165436 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7182823419570923 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7774606347084045 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16292738914489746 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15464095771312714 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45119571685791016 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 +jet/accuracy_1_of_1‚0.29942765831947327 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05945893004536629 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898105263710022 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940481781959534 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6672995686531067 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06262458860874176 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19301623106002808 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6863856315612793 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06243223696947098 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9958261847496033 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7247598171234131 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1703285127878189 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.621074378490448 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6685396432876587 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17994357645511627 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011179822497069836 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28685614466667175 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6555113196372986 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5475811958312988 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16669929027557373 7validation_average_jet_accuracy‚0.34889283776283264 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4329349398612976 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6727531552314758 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7132457494735718 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17764537036418915 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.448199350859921 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0399586521089077 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965658783912659 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17442511022090912 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9413429498672485 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7372869849205017 _runtime‚ 7987.31288446 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7110651135444641 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22194549441337585 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8599950075149536 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6840620040893555 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1391633003950119 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7686533331871033 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 &particle/f_score‚0.640720784664154 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20195814967155457 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1272965371608734 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5979758501052856 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6352010369300842 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36684074997901917 particle/accuracy_2_of_2‚1 _step‚395 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970111846923828 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877076148986816 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7277023196220398 *jet/accuracy_2_of_4‚0.8020323514938354 +jet/accuracy_4_of_4‚0.01628045178949833 % _timestamp‚1.7843277339374988e+09 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7281737327575684 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2360956072807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9951376914978027 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 particle/accuracy_1_of_1‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16709385812282562 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16083332896232605 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8755678534507751 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9969128370285034 +jet/accuracy_3_of_4‚0.09282653033733368 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6646806597709656 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48532480001449585 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6812702417373657 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1821439415216446 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9380581378936768 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7254852652549744 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8429516553878784 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6232031583786011 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4022408425807953 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8422147035598755 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867425560951233 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6296514868736267 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6328473091125488 *jet/accuracy_1_of_2‚0.7872010469436646 *particle/specificity‚0.911928653717041 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944680333137512 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028970610350370407 particle/accuracy_1_of_5‚1 trainer/global_step‚4799 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6746256947517395 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5777392387390137 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7393516302108765 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15358896553516388 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6476342082023621 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6157213449478149 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16669929027557373 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909444451332092 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8261895775794983 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8261895775794983 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8514739274978638 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16756819188594818 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020449616014957428 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6724276542663574 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18102063238620758 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020403003320097923 /particle/accuracy_3_of_4‚0.9965986609458923 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8475056886672974 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1550379991531372 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2462366372346878 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6916235685348511 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13896258175373077 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39104601740837097 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03802581503987312 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9969128370285034 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7790350317955017 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9202218055725098 *jet/accuracy_2_of_3‚0.5366390943527222 *jet/accuracy_1_of_4‚0.9710564613342285 +particle/sensitivity‚0.6231868863105774 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4657614529132843 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7377362251281738 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872538447380066 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8108655214309692 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06212696060538292 +jet/accuracy_3_of_3‚0.07175064831972122 (particle/accuracy‚0.8499231934547424 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995926022529602 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6239814162254333C7$£Šñ{ particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6724276542663574 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18102063238620758 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020403003320097923 /particle/accuracy_3_of_4‚0.9965986609458923 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8475056886672974 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1550379991531372 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2462366372346878 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6916235685348511 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13896258175373077 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39104601740837097 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03802581503987312 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9969128370285034 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7790350317955017 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9202218055725098 *jet/accuracy_2_of_3‚0.5366390943527222 *jet/accuracy_1_of_4‚0.9710564613342285 +particle/sensitivity‚0.6231868863105774 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4657614529132843 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7377362251281738 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872538447380066 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8108655214309692 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06212696060538292 +jet/accuracy_3_of_3‚0.07175064831972122 (particle/accuracy‚0.8499231934547424 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995926022529602 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6239814162254333 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9930832982063293 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6233391761779785 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.143853098154068 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18196415901184082 epoch‚11 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16920924186706543 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1604541540145874 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7411731481552124 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.621074378490448 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05945893004536629 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7389044761657715 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06187319755554199 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.14236336946487427 :Š©)T/"Purity/*FRt*SRqqt*FBt/event_purity‚0.28552260994911194 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5717385411262512 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3641650378704071 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12861086428165436 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7182823419570923 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7774606347084045 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16292738914489746 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15464095771312714 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45119571685791016 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 +jet/accuracy_1_of_1‚0.29942765831947327 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05945893004536629 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898105263710022 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940481781959534 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6672995686531067 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06262458860874176 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19301623106002808 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6863856315612793 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06243223696947098 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9958261847496033 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7247598171234131 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1703285127878189 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.621074378490448 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6685396432876587 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17994357645511627 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011179822497069836 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28685614466667175 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6555113196372986 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5475811958312988 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16669929027557373 7validation_average_jet_accuracy‚0.34889283776283264 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4329349398612976 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6727531552314758 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7132457494735718 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17764537036418915 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.448199350859921 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0399586521089077 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965658783912659 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17442511022090912 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9413429498672485 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7372869849205017 _runtime‚ 7987.31288446 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7110651135444641 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22194549441337585 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8599950075149536 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6840620040893555 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1391633003950119 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7686533331871033 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 &particle/f_score‚0.640720784664154 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20195814967155457 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1272965371608734 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5979758501052856 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6352010369300842 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36684074997901917 particle/accuracy_2_of_2‚1 _step‚395 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970111846923828 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877076148986816 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7277023196220398 *jet/accuracy_2_of_4‚0.8020323514938354 +jet/accuracy_4_of_4‚0.01628045178949833 % _timestamp‚1.7843277339374988e+09 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7281737327575684 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2360956072807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9951376914978027 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 particle/accuracy_1_of_1‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16709385812282562 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16083332896232605 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8755678534507751 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9969128370285034 +jet/accuracy_3_of_4‚0.09282653033733368 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6646806597709656 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48532480001449585 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6812702417373657 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1821439415216446 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9380581378936768 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7254852652549744 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8429516553878784 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6232031583786011 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4022408425807953 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8422147035598755 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867425560951233 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6296514868736267 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6328473091125488 *jet/accuracy_1_of_2‚0.7872010469436646 *particle/specificity‚0.911928653717041 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944680333137512 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028970610350370407 particle/accuracy_1_of_5‚1 trainer/global_step‚4799 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6746256947517395 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5777392387390137 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7393516302108765 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15358896553516388 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6476342082023621 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6157213449478149 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16669929027557373 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909444451332092 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8261895775794983 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8261895775794983 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8514739274978638 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16756819188594818 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020449616014957428 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075‹¦ à)ž‹‚(:“ ½Üê҇ʞ¬ gpu.0.gpu‚97.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 OÃr:Œ‚(:¯ ½ÜêÒÌï»® disk.sda1.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 16602.4921875 disk.rbd2.out‚58.78564453125) proc.memory.percent‚17.96443462371826* disk./.usagePercent‚39.200475516586344 disk.dm-0.in‚ 16602.4921875# disk.nvme0c0n1.in‚ 16602.4921875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19342.20068359375$ disk./.usageGB‚584.1154136657715 disk.sda1.in‚0% disk.nvme0n1.out‚66980.1884765625 disk.rbd1.in‚ 506.52734375 cpu‚25.588621867412716 disk.dm-1.in‚ 5.93359375 disk.sda2.out‚0$ memory_percent‚82.19854831695557" disk.dm-0.out‚66980.1884765625 disk.dm-1.out‚ 452.1640625 disk.drbd2165.in‚0 disk.sda3.out‚ 452.1640625 disk.sda.out‚ 452.1640625) proc.memory.availableMB‚ 2916.58984375 network.sent‚ 9.098253e+06 disk.sda3.in‚ 5.93359375' disk.nvme0c0n1.out‚66980.1884765625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2943.29296875 proc.cpu.threads‚17 network.recv‚ 2.058694e+06•Nü”‚(:‰ ÌÜêÒï• ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"@ñ=Ž‚(:² ÌÜêÒÊß̯ disk.drbd2165.out‚0 disk.sda.in‚ 5.93359375 proc.cpu.threads‚17" disk.dm-0.out‚66991.1337890625) proc.memory.availableMB‚ 2916.51953125 network.recv‚ 2.063248e+06* disk./.usagePercent‚39.200473724533595 disk.drbd2165.in‚0" disk.nvme0n1.in‚16603.44921875) proc.memory.percent‚17.96443462371826 network.sent‚ 9.155722e+06 disk.sda3.in‚ 5.93359375# disk.rbd1.out‚19351.81005859375 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚16603.44921875' disk.nvme0c0n1.out‚66991.1337890625% disk.nvme0n1.out‚66991.1337890625 disk.sda.out‚ 452.3671875 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2943.29296875 disk.dm-1.out‚ 452.3671875 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚16603.44921875$ disk./.usageGB‚584.1153869628906 disk.sda3.out‚ 452.3671875 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 disk.sda2.in‚0 disk.rbd0.in‚0$ memory_percent‚82.19897747039795 cpu‚25.588231141655584¶øj ;‚(:° ÛÜêÒÌÙ¡¯) disk./.usagePercent‚38.99121316604558 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19351.81005859375 disk.drbd2165.out‚0$ memory_percent‚82.19888210296631# proc.memory.rssMB‚ 2943.29296875# disk.dm-0.out‚67990.91455078125 proc.cpu.threads‚17& disk.nvme0n1.out‚67990.91455078125 disk.rbd0.in‚0 disk.sda3.in‚ 5.93359375 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 16741.484375 disk.drbd2165.in‚0$ disk./.usageGB‚580.9972534179688( disk.nvme0c0n1.out‚67990.91455078125 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0) proc.memory.availableMB‚ 2916.53515625) proc.memory.percent‚17.96443462371826 cpu‚25.587913592545878" disk.nvme0c0n1.in‚ 16741.484375 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.nvme0n1.in‚ 16741.484375 disk.sda.out‚ 453.04296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 453.04296875 disk.sda3.out‚ 453.04296875 disk.sda1.out‚0 network.sent‚ 9.16133e+06 network.recv‚ 2.066547e+06¬°¾çŸ‚(:” ÛÜêÒÿñÀ® gpu.0.gpu‚53.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00PTÑž‘‚(:“ êÜêÒîßå¬ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚294.438' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.88759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QrÜ“:’‚(:¯ êÜêÒ†Ò¦¯ disk.drbd2165.in‚0 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 453.5078125# disk.rbd1.out‚19351.87255859375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125( proc.memory.availableMB‚ 2916.3515625 disk.drbd2165.out‚0% disk.nvme0n1.out‚68012.3330078125 disk.sda.out‚ 453.5078125) proc.memory.percent‚17.96443462371826" disk.nvme0n1.in‚16742.31640625 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0$ memory_percent‚82.20000267028809 network.sent‚ 9.165841e+06$ disk.nvme0c0n1.in‚16742.31640625 disk.sda2.in‚0 cpu‚25.58750183492606 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.29296875$ disk./.usageGB‚580.9972534179688' disk.nvme0c0n1.out‚68012.3330078125 disk.sda.in‚ 5.93359375 disk.sda1.in‚0 disk.dm-1.out‚ 453.5078125" disk.dm-0.out‚68012.3330078125 network.recv‚ 2.069353e+06) disk./.usagePercent‚38.99121316604558 disk.dm-1.in‚ 5.93359375 disk.dm-0.in‚16742.31640625 proc.cpu.threads‚17û¨9”“‚(:‰ ùÜêÒŽ­® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÙZ9”‚(:® ùÜêÒ•¾–¯ disk.sda.out‚ 453.74609375 disk.rbd0.out‚0& proc.memory.availableMB‚ 2916.09375 disk.dm-1.in‚ 5.93359375$ disk.nvme0c0n1.in‚16742.31640625 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 cpu‚25.58712368073223$ disk./.usageGB‚580.9972991943359 disk.drbd2165.out‚0 disk.sda3.out‚ 453.74609375 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚16742.31640625% disk.nvme0n1.out‚68015.2568359375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125 disk.dm-1.out‚ 453.74609375 disk.rbd0.in‚0) proc.memory.percent‚17.96443462371826 disk.sda3.in‚ 5.93359375# disk.rbd1.out‚19351.87255859375 network.sent‚ 9.170173e+06 network.recv‚ 2.072856e+06" disk.dm-0.out‚68015.2568359375 disk.drbd2165.in‚0 disk.sda.in‚ 5.93359375' disk./.usagePercent‚38.991216238136 disk.dm-0.in‚16742.31640625' disk.nvme0c0n1.out‚68015.2568359375$ memory_percent‚82.20157623291016# proc.memory.rssMB‚ 2943.29296875ŸZº’•‚(:‡ ˆÝêÒšù¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚344.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨5Áu:–‚(:¯ ˆÝêÒ´Ü©° network.sent‚ 9.175245e+06 disk.sda.out‚ 454.16015625 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 454.16015625 proc.cpu.threads‚17 disk.sda2.in‚0# disk.rbd1.out‚19351.95068359375 cpu‚25.58680028143815* disk./.usagePercent‚38.991215214105864 disk.sda3.in‚ 5.93359375$ disk./.usageGB‚580.9972839355469! disk.nvme0n1.in‚ 16743.2734375% disk.nvme0n1.out‚68025.9423828125 disk.sda2.out‚0" disk.dm-0.out‚68025.9423828125 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 16743.2734375 disk.rbd1.in‚ 506.52734375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2943.29296875 network.recv‚ 2.076328e+06$ memory_percent‚82.20357894897461) proc.memory.percent‚17.96443462371826' disk.nvme0c0n1.out‚68025.9423828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 16743.2734375' proc.memory.availableMB‚ 2915.765625 disk.sda.in‚ 5.93359375 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 454.16015625¥§Ñ2—- _wandbruntime‚8071  _runtime‚8071.¤}UǘÁ % _timestamp‚1.7843278176520889e+09 _runtime‚8071.026773473 _step‚396 & lr-AdamW/pg1‚0.0002406757138932573 & lr-AdamW/pg2‚0.0002406757138932573 trainer/global_step‚4849L0œKÌ™Æ % _timestamp‚1.7843278176520889e+09 _runtime‚8071.026773473 _step‚396 & lr-AdamW/pg1‚0.0002406757138932573 & lr-AdamW/pg2‚0.0002406757138932573 trainer/global_step‚4849ŒœQë¯2š- _wandbruntime‚8071  _runtime‚8071«""6ÿ'›ùO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1394326794330112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1398278264077111 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4563397138432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399559 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376578543661056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12444770 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13783713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4555588928569 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚8071.031559248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1393724045176832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1393724019748864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1398888846078976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13783483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12444765 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4554244328247 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41387543 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389580 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1398890217641984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1394326820503552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5886536577024 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1394326820503552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26228248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5886504137728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42002043 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41387537 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚397 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42012016 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1398890217641984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚4849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4562051748864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1398279634105401 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843278176532567e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1394326794330112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4563397138432 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1398888846078976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13783483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocatòZ7EÇ$ions.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12444765 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4554244328247 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41387543 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389580 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1398890217641984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1394326820503552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5886536577024 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1394326820503552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26228248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5886504137728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42002043 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41387537 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚397 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42012016 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1398890217641984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚4849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4562051748864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1398279634105401 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843278176532567e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1394326794330112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4563397138432 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1394326794330112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1398278264077111 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4563397138432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399559 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376578543661056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12444770 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13783713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4555588928569 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚8071.031559248Œ!ƒ2- _wandbruntime‚8072  _runtime‚8072ø&H 'žƒN D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41395749 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5887744908288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1394604795060224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1398558439349935 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26233558 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376655661977600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5887712476160 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42020508 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4564322181120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26233786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416257 _runtime‚8072.421962018 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12447308 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7843278190470679e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13786473 _step‚398 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399169143414784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1394604795060224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42010521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406264 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚4849 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41395743 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399167771844608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1398557069321533 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416257 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41395743 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382543774977024ÝÙI'ŸˆN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26233558 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376655661977600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5887712476160 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42020508 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4564322181120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26233786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416257 _runtime‚8072.421962018 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12447308 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7843278190470679e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13786473 _step‚398 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399169143414784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1394604795060224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42010521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406264 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚4849 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41395743 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399167771844608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1398557069321533 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416257 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41395743 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382543774977024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41395749 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5887744908288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1394604795060224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1398558439349935 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926ŽSÜ2 - _wandbruntime‚8072  _runtime‚8072_õˆ ¡„ 3loss/SRqqt1/assignment_loss‚0.47329487359823974 2loss/SRqqt2/assignment_loss‚0.5361021347160898 2loss/FBt1/assignment_loss‚0.003859248485884705 .loss/FRt1/detection_loss‚1.388722814472604 0loss/FBt1/detection_loss‚0.45598633926720283 /loss/FRt1/assignment_loss‚0.814906412100294 0loss/FRt2/assignment_loss‚1.7766527712929483 1loss/SRqqt1/detection_loss‚0.6430395816785417 1loss/SRqqt2/detection_loss‚0.9957183195066821 %loss/total_loss‚8.233834869557679 trainer/global_step‚4849 _runtime‚8072.503278837 /loss/FRt2/detection_loss‚0.5904625254197776 epoch‚12 % _timestamp‚1.7843278191287293e+09 _step‚399 2loss/FBt2/assignment_loss‚0.022647559507381978 /loss/FBt2/detection_loss‚0.5324422895120313‹²ÉÖ¢‰ 3loss/SRqqt1/assignment_loss‚0.47329487359823974 2loss/SRqqt2/assignment_loss‚0.5361021347160898 2loss/FBt1/assignment_loss‚0.003859248485884705 .loss/FRt1/detection_loss‚1.388722814472604 0loss/FBt1/detection_loss‚0.45598633926720283 /loss/FRt1/assignment_loss‚0.814906412100294 0loss/FRt2/assignment_loss‚1.7766527712929483 1loss/SRqqt1/detection_loss‚0.6430395816785417 1loss/SRqqt2/detection_loss‚0.9957183195066821 %loss/total_loss‚8.233834869557679 trainer/global_step‚4849 _runtime‚8072.503278837 /loss/FRt2/detection_loss‚0.5904625254197776 epoch‚12 % _timestamp‚1.7843278191287293e+09 _step‚399 2loss/FBt2/assignment_loss‚0.022647559507381978 /loss/FBt2/detection_loss‚0.5324422895120313õ^*b”£‚(:‰ —ÝêÒàú߬ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.016' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Ð9¤‚(:® —ÝêÒ›©É®) disk./.usagePercent‚38.99123697474627 disk.sda2.out‚0$ disk.nvme0c0n1.in‚16879.59765625 cpu‚25.586897815619864 disk.dm-1.out‚ 454.3828125 disk.sda.in‚ 5.93359375 disk.dm-1.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375# memory_percent‚82.2073221206665 disk.rbd2.out‚58.78564453125" disk.dm-0.out‚68034.0166015625" disk.nvme0n1.in‚16879.59765625 disk.dm-0.in‚16879.59765625 disk.sda3.in‚ 5.93359375 network.sent‚ 9.18045e+06$ disk./.usageGB‚580.9976081848145 disk.sda.out‚ 454.3828125 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 454.3828125' disk.nvme0c0n1.out‚68034.0166015625) proc.memory.availableMB‚ 2915.15234375# proc.memory.rssMB‚ 2943.29296875% disk.nvme0n1.out‚68034.0166015625# disk.rbd1.out‚19351.95068359375 disk.drbd2165.in‚0) proc.memory.percent‚17.96443462371826 proc.cpu.threads‚17 network.recv‚ 2.07992e+06á½,’¥‚(:‰ ¦ÝêÒ½Ž£¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 4W9616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r.>O7¦‚(:¬ ¦ÝêÒŒ¤ï® network.sent‚ 9.244132e+06 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 454.796875( proc.memory.availableMB‚ 1943.4140625# proc.memory.rssMB‚ 2943.29296875 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 454.796875 disk.sda2.out‚0 disk.sda.out‚ 454.796875 disk.sda2.in‚0 disk.dm-0.in‚16879.59765625 disk.sda1.in‚0) proc.memory.percent‚17.96443462371826 proc.cpu.threads‚17 network.recv‚ 2.08503e+06 disk.rbd0.in‚0$ memory_percent‚88.13834190368652 cpu‚25.587104746523988# disk.rbd1.out‚19352.07568359375 disk.sda.in‚ 5.93359375 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375" disk.nvme0n1.in‚16879.59765625 disk.rbd2.out‚58.78564453125 disk.drbd2165.out‚0) disk./.usagePercent‚38.99113687580038$ disk./.usageGB‚580.9961166381836$ disk.nvme0c0n1.in‚16879.59765625' disk.nvme0c0n1.out‚68040.4716796875" disk.dm-0.out‚68040.4716796875% disk.nvme0n1.out‚68040.4716796875 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625_%ÿ”§‚(:‰ µÝêÒŠÿÖ® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¢Ùñ<¨‚(:± µÝêÒÒÛ±! disk.nvme0n1.in‚ 16881.0859375 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚68047.82861328125 disk.sda1.out‚0 disk.dm-1.out‚ 455.4609375 disk.sda.out‚ 455.4609375 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 16881.0859375) proc.memory.percent‚17.96443462371826 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 16881.0859375* disk./.usagePercent‚38.991252847213396 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375 cpu‚25.58747845792175# disk.rbd1.out‚19352.07568359375# disk.dm-0.out‚68047.82861328125# proc.memory.rssMB‚ 2943.29296875 network.sent‚ 9.249235e+06 network.recv‚ 2.088317e+06 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0 disk.sda1.in‚0$ disk./.usageGB‚580.9978446960449& disk.nvme0n1.out‚68047.82861328125 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 455.4609375 disk.rbd2.in‚0.015625$ memory_percent‚88.13846111297607) proc.memory.availableMB‚ 1943.39453125 proc.cpu.threads‚17™©çÞ’©‚(:‡ ÄÝêÒȾȬ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ù š5ª‚(:ª ÄÝêÒ¸Ïä¯ disk.rbd2.out‚58.78564453125 disk.sda.in‚ 5.93359375 disk.dm-0.in‚16881.91796875 disk.dm-0.out‚68056.48046875 disk.dm-1.in‚ 5.93359375( proc.memory.availableMB‚ 1943.3828125 network.sent‚ 9.253763e+06 network.recv‚ 2.091458e+06( disk./.usagePercent‚38.9912730718086 disk.sda.out‚ 455.80859375# disk.rbd1.out‚19352.15380859375 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚68056.48046875# memory_percent‚88.1385326385498 disk.rbd2.in‚0.015625$ disk./.usageGB‚580.9981460571289# disk.nvme0n1.out‚68056.48046875# proc.memory.rssMB‚ 2943.29296875 cpu‚25.58766733633918 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 455.80859375$ disk.nvme0c0n1.in‚16881.91796875" disk.nvme0n1.in‚16881.91796875 disk.rbd0.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.96443462371826 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 455.80859375 proc.cpu.threads‚17j?{”«‚(:‰ ÓÝêÒ—ýΫ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IïÑ6¬‚(:« ÓÝêÒÅË®) disk./.usagePercent‚38.99126743964285 disk.sda1.in‚0 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19352.15380859375) proc.memory.percent‚17.96443462371826 network.sent‚ 9.258931e+06 network.recv‚ 2.095301e+06$ disk./.usageGB‚580.9980621337891 disk.rbd2.in‚0.015625 disk.sda3.in‚ 5.93359375$ memory_percent‚88.13807964324951 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚456.0625 disk.sda2.in‚0# disk.dm-0.out‚68059.40576171875 disk.drbd2165.in‚0& disk.nvme0n1.out‚68059.40576171875) proc.memory.availableMB‚ 1943.45703125 disk.drbd2165.out‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚16977.19921875" disk.nvme0n1.in‚16977.19921875 disk.sda.out‚456.0625 disk.rbd1.in‚ 506.52734375 cpu‚25.587911948955952 disk.dm-1.out‚456.0625 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.29296875 disk.sda1.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚16977.19921875( disk.nvme0c0n1.out‚68059.40576171875©Ÿ(2­- _wandbruntime‚8151  _runtime‚8151/ïÏÇ®Á % _timestamp‚1.7843278983770623e+09 _runtime‚8151.751751384 _step‚400 & lr-AdamW/pg1‚0.0002399539496118574 & lr-AdamW/pg2‚0.0002399539496118574 trainer/global_step‚4899¦OÌ¯Æ trainer/global_step‚4899 % _timestamp‚1.7843278983770623e+09 _runtime‚8151.751751384 _step‚400 & lr-AdamW/pg1‚0.0002399539496118574 & lr-AdamW/pg2‚0.0002399539496118574V·Ûá2°- _wandbruntime‚8151  _runtime‚8151»q!ÿ'±ùO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1412836570078208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84235611 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4608367679488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12571920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380452864045056 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1412219960127309 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1412835198515200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1412218590099019 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1408226857009152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41797843 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609713069056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380452495953920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1412835198515200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609713069056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚4899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946824594432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4601826041677 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1408226857009152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1408226830835712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1408226830835712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41797837 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26494290 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _step‚401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386399688639488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5946790057984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4600481441355 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42427795 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42427795 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26494556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12571915 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1412836570078208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚8151.753077939•½ëV(²þO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13922636 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1407618134085632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13922375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1412218590099019 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1408226857009152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41797843 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609713069056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380452495953920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1412835198515200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609713069056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚4899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946824594432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4601826041677 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1408226857009152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1408226830835712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1408226830835712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41797837 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26494290 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _step‚401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386399688639488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5946790057984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4600481441355 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42427795 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42427795 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26494556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12571915 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1412836570078208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚8151.753077939 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1412836570078208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84235611 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4608367679488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12571920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380452864045056 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1412219960127309 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1412835198515200 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1412497395343441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242316 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42436273 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1413114124280832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42446260 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4609292715008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13925392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚4899 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42446260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26499855 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12574463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚402 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4610638111744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41806049 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1413115495851008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1408504857739264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1408504831565824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1412498765371843 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end¤ß¶5>/segment.small_pool.peak‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380529982361600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5947991123968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41806043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 % _timestamp‚1.7843278997760286e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26499594 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5948025653248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242316 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41806049 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1408504857739264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1408504831565824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1412498765371843 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380529982361600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5947991123968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41806043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 % _timestamp‚1.7843278997760286e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26499594 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5948025653248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242316 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41806049 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1412497395343441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242316 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42436273 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1413114124280832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42446260 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4609292715008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13925392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚4899 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42446260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26499855 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0’N)P2¶- _wandbruntime‚8153  _runtime‚8153I¨“6·€ 2loss/FBt1/assignment_loss‚0.009893417721925216 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5135244026651932 1loss/SRqqt1/detection_loss‚0.6647597306122529 /loss/FBt1/detection_loss‚0.6039831273335627 $loss/total_loss‚8.57249601039906 trainer/global_step‚4899 _runtime‚ 8153.23298638 0loss/FRt2/assignment_loss‚1.7519044481427026 1loss/FBt2/assignment_loss‚0.14932611408686358 /loss/FRt2/detection_loss‚0.5692706035913719 _step‚403 0loss/FRt1/assignment_loss‚0.6870228819858094 /loss/FRt1/detection_loss‚1.4127258792292103 1loss/SRqqt2/detection_loss‚0.9711972629237506 /loss/FBt2/detection_loss‚0.7445609521819279 $ _timestamp‚1.784327899858496e+09 2loss/SRqqt1/assignment_loss‚0.4943271899244893)_ Ô ¸… /loss/FBt1/detection_loss‚0.6039831273335627 $loss/total_loss‚8.57249601039906 trainer/global_step‚4899 _runtime‚ 8153.23298638 0loss/FRt2/assignment_loss‚1.7519044481427026 1loss/FBt2/assignment_loss‚0.14932611408686358 /loss/FRt2/detection_loss‚0.5692706035913719 _step‚403 0loss/FRt1/assignment_loss‚0.6870228819858094 /loss/FRt1/detection_loss‚1.4127258792292103 1loss/SRqqt2/detection_loss‚0.9711972629237506 /loss/FBt2/detection_loss‚0.7445609521819279 $ _timestamp‚1.784327899858496e+09 2loss/SRqqt1/assignment_loss‚0.4943271899244893 2loss/FBt1/assignment_loss‚0.009893417721925216 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5135244026651932 1loss/SRqqt1/detection_loss‚0.6647597306122529“1“m{”¹‚(:‰ âÝêÒ¬º¤® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚326.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ {=º‚(:² âÝêÒ®èÊ° disk.sda.in‚ 5.93359375& disk.nvme0n1.out‚68068.76318359375 disk.rbd0.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 9.264564e+06* disk./.usagePercent‚38.991261295462024 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 17091.4140625$ disk./.usageGB‚580.9979705810547 disk.sda3.in‚ 5.93359375 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125# disk.dm-0.out‚68068.76318359375 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 456.2421875 disk.sda3.out‚ 456.2421875( disk.nvme0c0n1.out‚68068.76318359375! disk.nvme0n1.in‚ 17091.4140625# disk.rbd1.out‚19352.21630859375 disk.rbd0.in‚0$ memory_percent‚88.13798427581787 cpu‚25.588226564788908# disk.nvme0c0n1.in‚ 17091.4140625 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1943.47265625) proc.memory.percent‚17.96443462371826 network.recv‚ 2.098563e+06 disk.sda.out‚ 456.2421875# proc.memory.rssMB‚ 2943.29296875s!D¬”»‚(:‰ ñÝêÒªñ§¬ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þøb @¼‚(:µ ñÝêÒˆ¯Û® disk.sda.out‚ 456.6328125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.out‚ 456.6328125 disk.rbd0.out‚0 disk.sda.in‚ 5.93359375) proc.memory.percent‚17.96443462371826# disk.dm-0.out‚68076.50830078125$ disk.nvme0c0n1.in‚17092.24609375" disk.nvme0n1.in‚17092.24609375& disk.nvme0n1.out‚68076.50830078125 cpu‚25.588473656426345 network.sent‚ 9.327175e+06* disk./.usagePercent‚38.991261295462024$ disk./.usageGB‚580.9979705810547 disk.rbd2.out‚58.78564453125 disk.sda3.out‚ 456.6328125# disk.rbd1.out‚19352.21630859375$ memory_percent‚88.13912868499756) proc.memory.availableMB‚ 1943.28515625 disk.dm-0.in‚17092.24609375 disk.drbd2165.in‚0 disk.sda3.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.29296875 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375( disk.nvme0c0n1.out‚68076.50830078125 disk.sda2.out‚0 network.recv‚ 2.103871e+06 disk.sda1.in‚0-Bj”½‚(:‰ €ÞêÒÉÀœ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÄm<9¾‚(:® €ÞêÒÄ…ð® proc.cpu.threads‚17 network.recv‚ 2.107237e+06) disk./.usagePercent‚38.99113431572504 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 456.90234375) proc.memory.availableMB‚ 1943.42578125 disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0# proc.memory.rssMB‚ 2943.29296875 disk.sda.in‚ 5.93359375$ disk./.usageGB‚580.9960784912109# disk.rbd1.out‚19352.29443359375 disk.rbd0.out‚0! disk.dm-0.out‚68082.701171875 disk.dm-1.in‚ 5.93359375& disk.nvme0c0n1.out‚68082.701171875# memory_percent‚88.1382703781128 disk.rbd2.in‚0.015625 cpu‚25.58870675405277 disk.rbd1.in‚ 506.52734375 disk.dm-0.in‚17092.24609375 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 456.90234375) proc.memory.percent‚17.96443462371826 network.sent‚ 9.33277e+06 disk.sda.out‚ 456.90234375$ disk.nvme0n1.out‚68082.701171875$ disk.nvme0c0n1.in‚17092.24609375" disk.nvme0n1.in‚17092.24609375ì¬>ž¿‚(:“ ÞêÒø¼ó¬ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ê‡-À‚(:¢ ÞêÒøïί disk.dm-1.in‚ 5.93359375# proc.memory.rssMB‚ 2943.29296875 network.sent‚ 9.337353e+06# disk./.usageGB‚580.996021270752$ disk.nvme0c0n1.out‚ 68091.8046875 disk.nvme0n1.in‚ 17093.203125" disk.nvme0n1.out‚ 68091.8046875& proc.memory.availableMB‚ 1943.40625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375 disk.sda.in‚ 5.93359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 17093.203125* disk./.usagePercent‚38.991130475612024 disk.rbd0.in‚0 disk.dm-0.in‚ 17093.203125 disk.sda.out‚ 457.19921875$ memory_percent‚88.13838958740234) proc.memory.percent‚17.96443462371826 cpu‚25.589026262184017# disk.rbd1.out‚19352.29443359375 network.recv‚ 2.110837e+06 disk.dm-0.out‚ 68091.8046875 disk.sda1.in‚0 disk.sda3.out‚ 457.19921875 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.out‚ 457.19921875 disk.rbd1.in‚ 506.52734375 disk.rbd2.out‚58.78564453125€èSPžÁ‚(:“ žÞêÒÁËú­ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.867' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.37340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½‚gE?‚(:´ žÞêÒõ¯×® disk.drbd2165.in‚0# disk.rbd1.out‚19352.35693359375# disk.dm-0.out‚68100.32568359375$ disk./.usageGB‚580.9960327148438" disk.nvme0n1.in‚17359.27734375& disk.nvme0n1.out‚68100.32568359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 5.93359375# proc.memory.rssMB‚ 2943.29296875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 cpu‚25.58923417053918 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚17359.27734375 disk.sda.in‚ 5.93359375 disk.sda1.out‚0$ memory_percent‚88.13846111297607) proc.memory.availableMB‚ 1943.39453125 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 network.sent‚ 9.342414e+06 network.recv‚ 2.114197e+06 disk.rbd2.in‚0.015625) proc.memory.percent‚17.96443462371826 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 457.4765625 disk.sda3.out‚ 457.4765625 disk.sda.out‚ 457.4765625( disk.nvme0c0n1.out‚68100.32568359375 disk.sda2.out‚0 disk.dm-0.in‚17359.27734375* disk./.usagePercent‚38.991131243634634útf{2Ã- _wandbruntime‚8232  _runtime‚8232ôÆ«ÉÄà trainer/global_step‚4949 % _timestamp‚1.7843279789389162e+09 _runtime‚8232.313537607 _step‚404 ' lr-AdamW/pg1‚0.00023922557492334269 ' lr-AdamW/pg2‚0.00023922557492334269Д/ÚÎÅÈ _runtime‚8232.313537607 _step‚404 ' lr-AdamW/pg1‚0.00023922557492334269 ' lr-AdamW/pg2‚0.00023922557492334269 trainer/global_step‚4949 % _timestamp‚1.7843279789389162e+09”fíͤ2Æ- _wandbruntime‚8232  _runtime‚8232 Ð_§Ñ"ÇùO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42863520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚384326816337920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6006522519552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚405 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421512222994432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1426781550951424 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071663 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426781550951424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426782922514432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6006559153152 % _timestamp‚1.7843279789399698e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14060342 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42853547 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384327184429056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1422126867341312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12699065 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42208137 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1426158916120927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1422126867341312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26759412 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390333743582208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4646718554463 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390333338857472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 I„.t/allocated_bytes.large_pool.allocated‚1422126893514752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26759140 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1422126893514752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14060075_˜Xa(ÈþO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390333338857472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1422126893514752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14060075 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42863520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚384326816337920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6006522519552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚405 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421512222994432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1426781550951424 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071663 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426781550951424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426782922514432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6006559153152 % _timestamp‚1.7843279789399698e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14060342 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42853547 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384327184429056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1422126867341312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12699065 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42208137 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1426158916120927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1422126867341312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26759412 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390333743582208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4646718554463 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008•SJ 2É- _wandbruntime‚8233  _runtime‚8233ndŠr'Ê‚N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421790104772608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12701613 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1422404868071424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088361 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _step‚406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421790079344640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _runtime‚8233.708318855 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1426439091393751 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4648986621143 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26764446 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426437721365349 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42872012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42862025 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4655608645632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427060476717056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42216343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12701608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088361 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42216343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14062838 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784327980333296e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6007741712384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390412007831552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384404670836736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078368 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14063098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4949 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42872012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42862025 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4655608645632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427060476717056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42216343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12701608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088361 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42216343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14062838 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784327980333296e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6007741712384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390412007831552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384404670836736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078368 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14063098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4949 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088361 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _step‚406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421790079344640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _runtime‚8233.708318855 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1426439091393751 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4648986621143 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26764446 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426437721365349 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175–ÏÓ‹Ò2Ì- _wandbruntime‚8233  _runtime‚82331Í Í„ _step‚407 2loss/SRqqt2/assignment_loss‚0.6563146359085495 /loss/FRt2/detection_loss‚0.5483428741602978 /loss/FBt1/detection_loss‚0.4706132984487744 0loss/FRt1/assignment_loss‚0.7739446102469055 2loss/FBt1/assignment_loss‚0.012867586602998336 3loss/FBt2/assignment_loss‚0.0003653065303392249 1loss/SRqqt2/detection_loss‚1.0761135497536196 % _timestamp‚1.7843279804161325e+09 2loss/SRqqt1/assignment_loss‚0.5331144525179475 /loss/FRt1/detection_loss‚1.3835881408088393 1loss/SRqqt1/detection_loss‚0.6113923853721815 epoch‚12 _runtime‚8233.790711328 /loss/FRt2/assignment_loss‚1.647195840254426 /loss/FBt2/detection_loss‚0.5690093567715057 %loss/total_loss‚8.282862037376384 trainer/global_step‚4949ëñ¯‚Ή _step‚407 2loss/SRqqt2/assignment_loss‚0.6563146359085495 /loss/FRt2/detection_loss‚0.5483428741602978 /loss/FBt1/detection_loss‚0.4706132984487744 0loss/FRt1/assignment_loss‚0.7739446102469055 2loss/FBt1/assignment_loss‚0.012867586602998336 3[¨ üloss/FBt2/assignment_loss‚0.0003653065303392249 1loss/SRqqt2/detection_loss‚1.0761135497536196 % _timestamp‚1.7843279804161325e+09 2loss/SRqqt1/assignment_loss‚0.5331144525179475 /loss/FRt1/detection_loss‚1.3835881408088393 1loss/SRqqt1/detection_loss‚0.6113923853721815 epoch‚12 _runtime‚8233.790711328 /loss/FRt2/assignment_loss‚1.647195840254426 /loss/FBt2/detection_loss‚0.5690093567715057 %loss/total_loss‚8.282862037376384 trainer/global_step‚4949—ËšížÏ‚(:“ ­ÞêÒø”ÿ« gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘X”:Ђ(:¯ ­ÞêÒ—žÇ¯" disk.nvme0c0n1.in‚ 17360.109375) disk./.usagePercent‚38.99126001542435 disk.rbd1.in‚ 506.52734375 disk.dm-1.in‚ 5.93359375) proc.memory.percent‚17.96443462371826 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚68107.95068359375 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19352.35693359375$ memory_percent‚88.13841342926025# proc.memory.rssMB‚ 2943.29296875 network.sent‚ 9.347616e+06 disk.sda2.in‚0# disk.dm-0.out‚68107.95068359375 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1943.40234375 disk.sda.out‚ 457.80078125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 17360.109375 network.recv‚ 2.117842e+06 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 457.80078125 disk.nvme0n1.in‚ 17360.109375 cpu‚25.5894600072202$ disk./.usageGB‚580.9979515075684( disk.nvme0c0n1.out‚68107.95068359375 disk.dm-1.out‚ 457.80078125ˆÀþè”Ñ‚(:‰ ¼ÞêÒðó§¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚341.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01t§ó2Ò‚(:§ ¼ÞêÒáÞ­# disk.nvme0n1.out‚68117.47265625 disk.sda2.out‚0 cpu‚25.58973252403924 network.recv‚ 2.122573e+06$ disk./.usageGB‚580.9984321594238" disk.nvme0n1.in‚17361.06640625 disk.rbd2.in‚0.015625# proc.memory.rssMB‚ 2943.29296875 network.sent‚ 9.41133e+06 disk.sda1.out‚0 disk.sda.out‚ 458.07421875 disk.rbd1.in‚ 506.52734375$ proc.memory.availableMB‚1943.375 disk.sda1.in‚0 disk.sda.in‚ 5.93359375 disk.drbd2165.in‚0 disk.dm-1.out‚ 458.07421875 disk.dm-0.out‚68117.47265625$ memory_percent‚88.13858032226562 disk.dm-0.in‚17361.06640625 disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚17361.06640625 disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 proc.cpu.threads‚17) disk./.usagePercent‚38.99129227237367 disk.sda3.out‚ 458.07421875 disk.rbd0.out‚0) proc.memory.percent‚17.96443462371826 disk.dm-1.in‚ 5.93359375% disk.nvme0c0n1.out‚68117.47265625 disk.sda2.in‚0# disk.rbd1.out‚19352.49755859375 disk.drbd2165.out‚0 ±‚?ŸÓ‚(:” ËÞêÒï™Å¬ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.642' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.528400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷C˜6Ô‚(:« ËÞêÒ™‚¯) proc.memory.percent‚17.96443462371826 disk.sda.out‚ 458.40625 disk.rbd2.out‚58.78564453125 disk.rbd2.in‚0.015625$ disk./.usageGB‚580.9979934692383" disk.dm-0.out‚68120.7158203125 disk.drbd2165.in‚0 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19352.49755859375 cpu‚25.589923764382203 disk.sda1.out‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1943.37109375 proc.cpu.threads‚17 disk.dm-0.in‚17361.06640625 disk.sda3.in‚ 5.93359375% disk.nvme0n1.out‚68120.7158203125 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 network.sent‚ 9.415962e+06 network.recv‚ 2.126046e+06 disk.sda3.out‚ 458.40625 disk.rbd0.out‚0$ memory_percent‚88.13860416412354 disk.sda1.in‚0' disk.nvme0c0n1.out‚68120.7158203125 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.29296875 disk.dm-1.in‚ 5.93359375) disk./.usagePercent‚38.99126283150723 disk.dm-1.out‚ 458.40625$ disk.nvme0c0n1.in‚17361.06640625" disk.nvme0n1.in‚17361.06640625ÀÞÐПÕ‚(:” ÚÞêÒ¤­¡¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%u}>Ö‚(:³ ÚÞêÒƒŽº¯ disk.rbd0.in‚0$ disk.nvme0c0n1.in‚17511.30078125( disk.nvme0c0n1.out‚68130.65478515625) proc.memory.availableMB‚ 1943.37890625 cpu‚25.590149913083177 proc.cpu.threads‚17) disk./.usagePercent‚38.99126641561271 disk.drbd2165.out‚0$ memory_percent‚88.13855648040771# proc.memory.rssMB‚ 2943.29296875 disk./.usageGB‚ 580.998046875 disk.rbd1.in‚ 506.52734375 network.sent‚ 9.421601e+06 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.dm-0.in‚17511.30078125 disk.drbd2165.in‚0# disk.dm-0.out‚68130.65478515625 disk.sda1.in‚0 disk.dm-1.out‚ 458.69921875) proc.memory.percent‚17.96443462371826 disk.sda.out‚ 458.69921875# disk.rbd1.out‚19352.56005859375 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 458.69921875" disk.nvme0n1.in‚17511.30078125 disk.sda2.in‚0& disk.nvme0n1.out‚68130.65478515625 network.recv‚ 2.129098e+06 disk.rbd2.in‚0.015625 disk.sda3.in‚ 5.93359375yê]˓ׂ(:ˆ éÞêÒ¢¼Ä¬ gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚323.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.733 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N(®).Ø‚(:£ éÞêÒᮜ¯# disk.rbd1.out‚19352.56005859375) proc.memory.availableMB‚ 1943.12890625# proc.memory.rssMB‚ 2943.29296875 disk.dm-0.out‚68138.77734375 disk.drbd2165.in‚0 disk.sda2.out‚0$ disk./.usageGB‚580.9981575012207 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.sda.out‚ 459.203125) proc.memory.percent‚17.96443462371826 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375# disk.nvme0c0n1.in‚ 17512.2578125% disk.nvme0c0n1.out‚68138.77734375 cpu‚25.590469627322907( disk./.usagePercent‚38.9912738398312! disk.nvme0n1.in‚ 17512.2578125 disk.rbd0.in‚0 disk.dm-1.out‚ 459.203125 disk.sda.in‚ 5.93359375$ memory_percent‚88.14008235931396 disk.sda2.in‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 17512.2578125 disk.sda3.out‚ 459.203125 network.sent‚ 9.426131e+06 network.recv‚ 2.13269e+06 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375# disk.nvme0n1.out‚68138.77734375 disk.rbd1.in‚ 506.52734375-PìNžÙ‚(:“ øÞêÒ³›¸­ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.304' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.86079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÙC2Ú‚(:§ øÞêÒóÝ®¯ disk.sda2.out‚0# disk.rbd1.out‚19352.63818359375 network.sent‚ 9.431249e+06) disk./.usagePercent‚38.99127076774079 disk.sda1.out‚0$ memory_percent‚88.14003467559814 disk.dm-0.in‚ 17512.2578125 disk.sda1.in‚0% disk.nvme0c0n1.out‚68145.07421875 disk.sda.out‚ 459.4609375 disk.rbd2.out‚58.78564453125 disk.drbd2165.in‚0 cpu‚25.590665910602212 network.recv‚ 2.13585e+06 disk.dm-1.out‚ 459.4609375 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 459.4609375 disk.rbd0.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 17512.2578125# disk.nvme0n1.out‚68145.07421875 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 5.93359375! disk.nvme0n1.in‚ 17512.2578125$ disk./.usageGB‚580.9981117248535 disk.dm-0.out‚68145.07421875 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.29296875 proc.cpu.threads‚17 disk.rbd1.in‚ 506.52734375) proc.memory.availableMB‚ 1943.13671875) proc.memory.percent‚17.96443462371826‘¨&‚2Û- _wandbruntime‚8312  _runtime‚8312àTjÉÜà _runtime‚8312.438626216 _step‚408 ' lr-AdamW/pg1‚0.00023849063660985828 ' lr-AdamW/pg2‚0.00023849063660985828 trainer/global_step‚4999 % _timestamp‚1.7843280590639496e+09¶ÚØ4ÎÝÈ trainer/global_step‚4999 % _timestamp‚1.7843280590639496e+09 _runtime‚8312.438626216 _step‚408 ' lr-AdamW/pg1‚0.00023849063660985828 ' lr-AdamW/pg2‚0.00023849063660985828˜ 1]2Þ- _wandbruntime‚8312  _runtime‚8312dƒ#¢ÿ'ßùO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12826215 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4700999540736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4692955667571 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42618437 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4702344930304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42618437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _step‚409 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1436026903846912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42618443 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1440099242142835 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394267022921728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6065886199808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14197725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394267427646464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8312.439915029 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1440727903387648 trainer/global_step‚4999 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843280590651412e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702344930304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979$¨¥(àþO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394267427646464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8312.439915029 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1440727903387648 trainer/global_step‚4999 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843280590651412e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702344930304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12826215 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4700999540736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4692955667571 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42618437 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4702344930304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42618437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _step‚409 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1436026903846912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42618443 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1440099242142835 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394267022921728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6065886199808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14197725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154™þôц2á- _wandbruntime‚8313  _runtime‚83133 *—`âƒN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1436304904577024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1435684193681408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695223734251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914420 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4703269972992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394346097888256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4693879133817 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42626649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27028965 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1436304930750464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚8313.836369428 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6067106667520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43297764 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43287777 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14200207 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388278991220736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12828763 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27029226 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1441008200723456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1441008200723456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914420 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703269972992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4701924576256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924413 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843280604614773e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6067106667520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43297764 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43287777 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14200207 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388278991220736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12828763 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27029226 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1441008200723456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1441008200723456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914420 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703269972992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4701924576256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924413 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843280604614773e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394346097888256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4693879133817 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42626649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27028965 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1436304930750464šbmzY2ä- _wandbruntime‚8313  _runtime‚8313O^ å… 0loss/FRt1/assignment_loss‚0.7347464730592287 2loss/SRqqt2/assignment_loss‚0.5049278683062377 /loss/FRt1/detection_loss‚1.4215454026327834 1loss/SRqqt1/detection_loss‚0.5780219586614693 0loss/FBt1/detection_loss‚0.44068456860259175 trainer/global_step‚4999 0loss/FRt2/assignment_loss‚1.7327785644502867 2loss/FBt1/assignment_loss‚0.014255338279010559 2loss/FBt2/assignment_loss‚0.010844171348409582 1loss/SRqqt2/detection_loss‚0.8873093619663592 /loss/FBt2/detection_loss‚0.9643312955275178 %loss/total_loss‚8.369609523995132 epoch‚12 % _timestamp‚1.7843280605437167e+09 2loss/SRqqt1/assignment_loss‚0.5015031102534797 /loss/FRt2/detection_loss‚0.5786614109077575 _runtime‚8313.918246979 _step‚411î/˜ìæŠ _step‚411 0loss/FRt1/assignment_loss‚0.7347464730592287 2loss/SRqqt2/assignment_loss‚0.5049278683062377 /loss/FRt1/detection_loss‚1.4215454026327834 1loss/SRqqt1/detection_loss‚0.5780219586614693 0loss/FBt1/detection_loss‚0.44068456860259175 trainer/global_step‚4999 0loss/FRt2/assignment_loss‚1.7327785644502867 2loss/FBt1/assignment_loss‚0.014255338279010559 2loss/FBt2/assignment_loss‚0.010844171348409582 1loss/SRqqt2/detection_loss‚0.8873093619663592 /loss/FBt2/detection_loss‚0.9643312955275178 %loss/total_loss‚8.369609523995132 epoch‚12 % _timestamp‚1.7843280605437167e+09 2loss/SRqqt1/assignment_loss‚0.5015031102534797 /loss/FRt2/detection_loss‚0.5786614109077575 _runtime‚8313.918246979›y㑾”ç‚(:‰ ‡ßêÒÙŒ³¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÇÏ+5è‚(:ª ‡ßêÒ‹î¯ disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0$ memory_percent‚88.13402652740479 proc.cpu.threads‚17 network.recv‚ 2.139491e+06 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.out‚68154.16796875$ disk./.usageGB‚580.9981269836426 disk.sda.out‚ 459.7578125 disk.sda2.in‚0# disk.rbd1.out‚19352.63818359375 disk.dm-1.out‚ 459.7578125 disk.dm-0.out‚68154.16796875 disk.sda1.in‚0% disk.nvme0c0n1.out‚68154.16796875" disk.nvme0n1.in‚17513.08984375) proc.memory.availableMB‚ 1944.12109375) disk./.usagePercent‚38.99127179177093 disk.dm-0.in‚17513.08984375# proc.memory.rssMB‚ 2943.29296875 network.sent‚ 9.436303e+06 disk.rbd0.in‚0 disk.sda3.out‚ 459.7578125$ disk.nvme0c0n1.in‚17513.08984375 cpu‚25.59080393277491 disk.sda3.in‚ 5.93359375) proc.memory.percent‚17.96443462371826 disk.rbd2.out‚58.78564453125À…¡Çžé‚(:“ –ßêҵ‹¬ gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W *ê‚(:Ÿ –ßêÒîáß®) disk./.usagePercent‚38.99353336232841 disk.sda.out‚ 461.515625 disk.dm-1.out‚ 461.515625$ disk./.usageGB‚581.0318260192871# disk.nvme0c0n1.out‚ 68176.796875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚17666.57421875( proc.memory.availableMB‚ 1944.1171875# proc.memory.rssMB‚ 2943.29296875$ disk.nvme0c0n1.in‚17666.57421875 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 proc.cpu.threads‚17 network.sent‚ 9.499466e+06" disk.nvme0n1.in‚17666.57421875! disk.nvme0n1.out‚ 68176.796875 disk.sda2.out‚0# disk.rbd1.out‚19352.70068359375 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# memory_percent‚88.1340503692627) proc.memory.percent‚17.96443462371826 network.recv‚ 2.144729e+06 disk.sda.in‚ 5.93359375 disk.sda1.in‚0 cpu‚25.59109175137648 disk.rbd1.in‚ 506.52734375 disk.rbd2.out‚58.78564453125 disk.dm-0.out‚ 68176.796875 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 461.515625ñ휔ë‚(:‰ ¥ßêÒ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üæ–Ë;ì‚(:° ¥ßêÒ—Ù“¯ disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 462.02734375" disk.nvme0n1.in‚17666.57421875 disk.rbd2.in‚0.015625! disk.dm-0.out‚68180.142578125$ memory_percent‚88.13402652740479 cpu‚25.59131398096509 disk.sda2.in‚0 disk.rbd2.out‚58.78564453125 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 462.02734375) proc.memory.percent‚17.96443462371826$ disk./.usageGB‚581.0318870544434 disk.rbd0.out‚0) proc.memory.availableMB‚ 1944.12109375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚17666.57421875& disk.nvme0c0n1.out‚68180.142578125) disk./.usagePercent‚38.99353745844895 disk.sda1.out‚0 disk.sda.in‚ 5.93359375 disk.sda.out‚ 462.02734375# proc.memory.rssMB‚ 2943.29296875$ disk.nvme0n1.out‚68180.142578125 network.sent‚ 9.504666e+06 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19352.70068359375 proc.cpu.threads‚17 network.recv‚ 2.148394e+06 disk.dm-0.in‚17666.57421875¸Èª–5í‚(:ª ´ßêÒöׇ® disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 disk.dm-1.out‚ 462.66015625 network.sent‚ 9.509721e+06' disk.nvme0c0n1.out‚68220.2607421875 disk.sda2.in‚0" disk.dm-0.out‚68220.2607421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 5.93359375) proc.memory.percent‚17.96443462371826$ disk./.usageGB‚581.0316619873047 disk.nvme0n1.in‚ 17667.40625# disk.rbd1.out‚19352.77880859375$ memory_percent‚88.13331127166748 proc.cpu.threads‚17 disk.sda3.in‚ 5.93359375! disk.nvme0c0n1.in‚ 17667.40625% disk.nvme0n1.out‚68220.2607421875# proc.memory.rssMB‚ 2943.29296875 cpu‚25.59151513272389 disk.sda.in‚ 5.93359375 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1944.23828125 network.recv‚ 2.151704e+06 disk.sda3.out‚ 462.66015625 disk.rbd1.in‚ 506.52734375 disk.dm-0.in‚ 17667.40625) disk./.usagePercent‚38.99352235400443 disk.sda.out‚ 462.66015625ßµÆÄ”î‚(:‰ ´ßêÒÏù‹Ù gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸAC&Ÿï‚(:” ÃßêÒ½¹×« gpu.0.gpu‚39.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Zö¯1ð‚(:¦ ÃßêÒ²‹À® disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 463.14453125 disk.sda3.out‚ 463.14453125# disk.rbd1.out‚19352.77880859375 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375$ disk./.usageGB‚581.0317344665527 disk.sda.out‚ 463.14453125 disk.rbd0.in‚0 disk.dm-0.in‚ 17667.40625 disk.sda1.in‚0% disk.nvme0c0n1.out‚68223.44921875) proc.memory.percent‚17.96443462371826 network.sent‚ 9.514831e+06 network.recv‚ 2.155388e+06$ memory_percent‚88.13331127166748# proc.memory.rssMB‚ 2943.29296875 disk.nvme0n1.in‚ 17667.40625 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.out‚68223.44921875# disk.nvme0n1.out‚68223.44921875* disk./.usagePercent‚38.993527218147584 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1944.23828125 disk.rbd2.in‚0.015625! disk.nvme0c0n1.in‚ 17667.40625 cpu‚25.591884189549916 proc.cpu.threads‚17- %á2ñ- _wandbruntime‚8392  _runtime‚8392ê-¿vÆòÀ ' lr-AdamW/pg1‚0.00023774918187511872 ' lr-AdamW/pg2‚0.00023774918187511872 trainer/global_step‚5049 # _timestamp‚1.78432813951895e+09 _runtime‚ 8392.89355934 _step‚412€¥©yËóÅ _runtime‚ 8392.89355934 _step‚412 ' lr-AdamW/pg1‚0.00023774918187511872 ' lr-AdamW/pg2‚0.00023774918187511872 trainer/global_step‚5049 # _timestamp‚1.78432813951895e+09œ±9Ž>2ô- _wandbruntime‚8392  _runtime‚8392†Çý‚õøO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454675627386880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748660860928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743767 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27288070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454674255823872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12953370 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43715024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747315471360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43028737 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1449926940352512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398200816732160 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1449300400812032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 8392.89484634 _step‚413 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1449926940352512 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392075457105920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748660860928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14334700 trainer/global_step‚5049 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739192780679 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1449926966525952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43028737 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1454039568164743 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12953365 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27287791 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14334426 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733788 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43028743 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454675627386880 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454674255823872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6125396259840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739192780679 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1449926966525952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43028737 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1454039568164743 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12953365 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27287791 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14334426 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733788 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43028743 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454675627386880 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454674255823872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6125396259840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454674255823872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12953370 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43715024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747315471360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43028737 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1449926940352512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398200816732160 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1449300400812032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 8392.89484634 _step‚413 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1449926940352512 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392075457105920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748660860928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14334700 trainer/global_step‚5049 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8394.289429205 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450204967256064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843281409145808e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12955908 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43723516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4748240506880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14337435 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1450204967256064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1450204941082624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14337173 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1454953181589504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450204941082624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749585903616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27293348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12955913 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392152943513600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1454319743437567 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4741460847359 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1449578282590208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27293081 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760465 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43723516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8394.289429205 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450204967256064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843281409145808e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12955908 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43723516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4748240506880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14337435 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1450204967256064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1450204941082624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14337173 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1454953181589504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450204941082624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749585903616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27293348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12955913 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.l™Ëüë arge_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392152943513600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1454319743437567 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4741460847359 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1449578282590208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27293081 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760465 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43723516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27552158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402134542345216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1463194489720832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚5099 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4794976791552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13080515 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1467979894186651 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1463827003031552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4794976791552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468620608260096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468621979823104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1463194464292864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1467981264214941 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _step‚417 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468621979823104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44140776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44130803 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395950145581056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468620608260096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4793631401984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43439037 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1463826976858112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4793631401984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27552434 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13080520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1463827003031552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43439043 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843282193320413e+09*°(ŒþO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44130803 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395950145581056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468620608260096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4793631401984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43439037 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1463826976858112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4793631401984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27552434 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13080520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1463827003031552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43439043 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843282193320413e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579819 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27552158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402134542345216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1463194489720832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚5099 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4794976791552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13080515 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1467979894186651 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1463827003031552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4794976791552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468620608260096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468621979823104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1463194464292864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1467981264214941 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _step‚417 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468621979823104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44140776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820¡xOÉÜ"- _wandbruntime‚8474  !èi®_runtime‚8474½6x™ 'ŽƒN TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843282207271922e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1464105003761664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27557700 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13083058 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4795901834240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402213211089920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468900905595904 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786353360033 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1468258699431073 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1463472346071040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396027631988736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚418 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596517 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13083063 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚8474.102020677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1464104977588224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4794556437504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6185947192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1463472371499008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4794556437504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1464104977588224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596517 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27557429 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43447249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44139281 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396027263897600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185983818752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43447243 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4795901834240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44149268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787697960467 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1464105003761664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14474637 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14474371 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1468260069459475 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44149268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627}þž'ˆN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396027631988736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚418 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596517 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13083063 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚8474.102020677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1464104977588224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4794556437504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6185947192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1463472371499008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4794556437504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1464104977588224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596517 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27557429 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43447249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44139281 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396027263897600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185983818752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43447243 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4795901834240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44149268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787697960467 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1464105003761664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14474637 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14474371 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1468260069459475 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44149268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843282207271922e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1464105003761664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27557700 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13083058 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4795901834240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402213211089920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468900905595904 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786353360033 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1468258699431073 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1463472346071040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¢u—4ö2- _wandbruntime‚8474  _runtime‚8474d}‡*‘ % _timestamp‚1.7843282208089426e+09 2loss/SRqqt1/assignment_loss‚0.6107947868563767 /loss/FRt1/detection_loss‚1.4547355723447475 0loss/SRqqt2/detection_loss‚0.880271436016744 0loss/FBt1/detection_loss‚0.38033291684185566 /loss/FBt2/detection_loss‚0.7078788843378425 epoch‚12 _runtime‚8474.183507409 0loss/FRt1/assignment_loss‚0.7591552990015689 0loss/FRt2/assignment_loss‚1.7710081844670431 2loss/SRqqt2/assignment_loss‚0.5356888576518836 1loss/FBt1/assignment_loss‚0.01495526210894817 %loss/total_loss‚8.338323398531841 _step‚419 1loss/FBt2/assignment_loss‚0.00772604654981933 1loss/SRqqt1/detection_loss‚0.6313091056170143 trainer/global_step‚5099 .loss/FRt2/detection_loss‚0.584467046737998Ôßj… ’† 0loss/FBt1/detection_loss‚0.38033291684185566 /loss/FBt2/detection_loss‚0.7078788843378425 epoch‚12 _runtime‚8474.183507409 0loss/FRt1/assignment_loss‚0.7591552990015689 0loss/FRt2/assignment_loss‚1.7710081844670431 2loss/SRqqt2/assignment_loss‚0.5356888576518836 1loss/FBt1/assignment_loss‚0.01495526210894817 %loss/total_loss‚8.338323398531841 _step‚419 1loss/FBt2/assignment_loss‚0.00772604654981933 1loss/SRqqt1/detection_loss‚0.6313091056170143 trainer/global_step‚5099 .loss/FRt2/detection_loss‚0.584467046737998 % _timestamp‚1.7843282208089426e+09 2loss/SRqqt1/assignment_loss‚0.6107947868563767 /loss/FRt1/detection_loss‚1.4547355723447475 0loss/SRqqt2/detection_loss‚0.880271436016744£[øÔk“‚(:’ àêÒÖæê« gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢úT²4”‚(:© àêÒØ÷د network.recv‚ 2.176917e+06 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚464.8125 disk.sda.out‚464.8125 disk.drbd2165.out‚0) disk./.usagePercent‚38.99352158598183$ disk./.usageGB‚581.0316505432129 disk.dm-0.in‚17971.39453125# disk.dm-0.out‚68269.91845703125 disk.sda3.in‚ 5.93359375# proc.memory.rssMB‚ 2943.29296875) proc.memory.percent‚17.96443462371826 disk.rbd2.in‚0.015625 disk.dm-1.out‚464.8125 disk.sda2.in‚0 cpu‚25.59110974110766 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚17971.39453125" disk.nvme0n1.in‚17971.39453125& disk.nvme0n1.out‚68269.91845703125 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19352.98193359375 disk.rbd2.out‚58.78564453125( disk.nvme0c0n1.out‚68269.91845703125 disk.sda2.out‚0# memory_percent‚88.1373643875122 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0 network.sent‚ 9.603399e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1943.57421875\nûÒ”•‚(:‰ ¬àêÒ̧¶¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.8206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0; —=–‚(:² ¬àêÒ¹üå® disk.sda2.in‚0 disk.sda3.in‚ 5.93359375& disk.nvme0n1.out‚68279.88330078125 disk.sda.in‚ 5.93359375# disk.dm-0.out‚68279.88330078125 disk.drbd2165.out‚0 disk.sda.out‚ 465.07421875) proc.memory.availableMB‚ 1943.23046875$ disk./.usageGB‚581.0317077636719 disk.sda3.out‚ 465.07421875# disk.rbd1.out‚19353.06005859375 network.recv‚ 2.182161e+06 disk.sda2.out‚0 disk.dm-0.in‚ 17972.3515625 disk.sda1.in‚0 network.sent‚ 9.666557e+06 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 cpu‚25.59071990364336! disk.nvme0n1.in‚ 17972.3515625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 465.07421875# memory_percent‚88.1394624710083) proc.memory.percent‚17.96443462371826) disk./.usagePercent‚38.99352542609484# disk.nvme0c0n1.in‚ 17972.3515625 disk.sda1.out‚0( disk.nvme0c0n1.out‚68279.88330078125 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375# proc.memory.rssMB‚ 2943.29296875 proc.cpu.threads‚17Í †¨”—‚(:‰ »àêÒ´ÿ¥® gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚343.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­{±4˜‚(:© »àêÒ¿¹ö¯ network.recv‚ 2.185761e+06 disk.dm-1.out‚ 465.41015625" disk.nvme0n1.out‚ 68287.6640625 disk.sda.in‚ 5.93359375 disk.sda.out‚ 465.41015625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 9.671126e+06 disk.sda3.out‚ 465.41015625" disk.nvme0n1.in‚17973.18359375 cpu‚25.590406787283154 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk./.usageGB‚581.0317726135254 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.sda1.in‚0$ memory_percent‚88.13753128051758 disk.dm-0.out‚ 68287.6640625) disk./.usagePercent‚38.99352977822292$ disk.nvme0c0n1.out‚ 68287.6640625 disk.sda2.out‚0# disk.rbd1.out‚19353.06005859375 disk.rbd2.out‚58.78564453125 disk.dm-1.in‚ 5.93359375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 1943.546875) proc.memory.percent‚17.96443462371826 disk.sda3.in‚ 5.93359375 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.29296875$ disk.nvme0c0n1.in‚17973.18359375 disk.dm-0.in‚17973.18359375!â×qŸ™‚(:” ÊàêÒ׆¢¬ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.915800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Xž0š‚(:¥ ÊàêÒÜÏ„¯ disk.rbd0.out‚0$ disk./.usageGB‚581.0287055969238 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19353.12255859375 disk.rbd2.in‚0.015625 proc.cpu.threads‚17 disk.sda1.out‚0# disk.nvme0n1.out‚68293.91015625 disk.sda.in‚ 5.93359375' proc.memory.availableMB‚ 1943.296875" disk.nvme0n1.in‚17973.18359375 disk.rbd0.in‚0 disk.dm-1.out‚ 465.546875 disk.sda.out‚ 465.546875 disk.dm-0.in‚17973.18359375 disk.dm-1.in‚ 5.93359375 disk.rbd2.out‚58.78564453125# proc.memory.rssMB‚ 2943.29296875$ disk.nvme0c0n1.in‚17973.18359375% disk.nvme0c0n1.out‚68293.91015625$ memory_percent‚88.13905715942383) proc.memory.percent‚17.96443462371826 disk.sda3.out‚ 465.546875 network.sent‚ 9.676197e+06( disk./.usagePercent‚38.9933239481654 disk.drbd2165.in‚0 disk.sda3.in‚ 5.93359375 cpu‚25.590048789422514 network.recv‚ 2.189068e+06 disk.dm-0.out‚68293.91015625 disk.sda1.in‚06¡Î›‚(:’ ÙàêÒø“Ó¬ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚297.04' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.40800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ­#4œ‚(:© ÙàêÒØŒ„¯# disk./.usageGB‚581.028865814209 disk.sda3.in‚ 5.93359375 disk.sda2.out‚0 cpu‚25.58971578407423 disk.sda3.out‚ 465.76171875! disk.nvme0c0n1.in‚ 18096.65625 disk.nvme0n1.in‚ 18096.65625# proc.memory.rssMB‚ 2943.29296875 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375# disk.dm-0.out‚68302.36962890625$ memory_percent‚88.13753128051758 disk.dm-1.out‚ 465.76171875& disk.nvme0n1.out‚68302.36962890625 disk.sda.in‚ 5.93359375 disk.sda2.in‚0# disk.rbd1.out‚19353.12255859375 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1943.546875) proc.memory.percent‚17.96443462371826 proc.cpu.threads‚17 network.sent‚ 9.681301e+06) disk./.usagePercent‚38.99333470048184 disk.sda.out‚ 465.76171875 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚68302.36962890625 disk.dm-0.in‚ 18096.65625 network.recv‚ 2.19271e+06ímŸ‚(:” èàêÒ¬Ž®¬ gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ßÌ1ž‚(:¦ èàêÒ¦Åä­ disk.drbd2165.out‚0 disk.sda.in‚ 5.93359375 cpu‚25.589333391243077 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 466.078125 disk.sda.out‚ 466.078125 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.29296875 disk.dm-1.in‚ 5.93359375 disk.nvme0n1.in‚ 18096.65625 disk.sda2.out‚0$ memory_percent‚88.13753128051758 network.recv‚ 2.195936e+06$ disk./.usageGB‚581.0287780761719 disk.dm-0.in‚ 18096.65625! disk.nvme0c0n1.in‚ 18096.65625 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.out‚58.78564453125# disk.dm-0.out‚68307.33056640625( disk.nvme0c0n1.out‚68307.33056640625# disk.rbd1.out‚19353.20068359375 disk.rbd0.out‚0) proc.memory.percent‚17.96443462371826) disk./.usagePercent‚38.99332881230855 disk.sda1.out‚0 disk.dm-1.out‚ 466.078125& disk.nvme0n1.out‚68307.33056640625' proc.memory.availableMB‚ 1943.546875 proc.cpu.threads‚17 network.sent‚ 9.686368e+06 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0xnáà2Ÿ- _wandbruntime‚8552  _runtime‚8552Áûë†É Ã _runtime‚8552.458102997 _step‚420 ' lr-AdamW/pg1‚0.00023624691404636143 ' lr-AdamW/pg2‚0.00023624691404636143 trainer/global_step‚5149 % _timestamp‚1.7843282990834467e+09âDÚ×Î¡È _runtime‚8552.458102997 _step‚420 ' lr-AdamW/pg1‚0.00023624691404636143 ' lr-AdamW/pg2‚0.00023624691404636143 trainer/global_step‚5149 % _timestamp‚1.7843282990834467e+09¤<øì2¢- _wandbruntime‚8552  _runtime‚8552ÿvþ£ùO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27816977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1482568332259328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1482566960696320 trainer/global_step‚5149 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481920220208559 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4833011607217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399824097873920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1477727013363712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.M>ñá!small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1481921590236849 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44566528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1477727039537152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43849343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44566528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6244551202816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44556555 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405892 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13207670 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4831667006895 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820ó¹Ó(¤þO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27816977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1482568332259328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1482566960696320 trainer/global_step‚5149 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481920220208559 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4833011607217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399824097873920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1477727013363712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1481921590236849 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44566528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1477727039537152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43849343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44566528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6244551202816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44556555 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405892 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13207670 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4831667006895 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820¥2ä2¥- _wandbruntime‚8553  _runtime‚8553 …û'¦‚N G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88432569 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1478005040267264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1477366434979840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14612289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44575020 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432569 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4840872368128 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1482847258032128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44565033 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1478005014093824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1478005040267264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5149 _runtime‚ 8553.85456235 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832590473141 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27822257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4842217764864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4833935073575 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43857543 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1482847258032128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482200395481383 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4842217764864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399901952372736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399901584281600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44565033 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚11757+†¾§‡N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43857549 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚422 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432569 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4840872368128 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1482847258032128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44565033 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1478005014093824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1478005040267264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5149 _runtime‚ 8553.85456235 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832590473141 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27822257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4842217764864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4833935073575 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43857543 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1482847258032128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482200395481383 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4842217764864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399901952372736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399901584281600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44565033 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88432569 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1478005040267264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1477366434979840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14612289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44575020 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Fâ7²‚(:¬ ¤áêÒ‚Æ­ disk.drbd2165.out‚0# memory_percent‚88.1364107131958 disk.sda.in‚ 5.93359375 disk.sda1.in‚0$ disk./.usageGB‚581.8026542663574" disk.nvme0n1.in‚18316.30859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 9.764967e+06 disk.sda.out‚ 467.28125 disk.dm-1.in‚ 5.93359375) proc.memory.availableMB‚ 1943.73046875# proc.memory.rssMB‚ 2943.29296875 cpu‚25.58790345870351 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚18316.30859375 proc.cpu.threads‚17 disk.sda3.out‚ 467.28125 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19353.40380859375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625) disk./.usagePercent‚39.04526429275727 disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚18316.30859375& disk.nvme0n1.out‚68368.33837890625 disk.drbd2165.in‚0 disk.dm-1.out‚ 467.28125 network.recv‚ 2.211832e+06( disk.nvme0c0n1.out‚68368.33837890625# disk.dm-0.out‚68368.33837890625) proc.memory.percent‚17.96443462371826«mž³‚(:“ ³áêÒÕæÏ« gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*ÍÒZ0´‚(:¥ ³áêÒ´ðï­ disk.nvme0n1.in‚ 18317.265625$ disk.nvme0n1.out‚68377.224609375 disk.sda3.in‚ 5.93359375& proc.memory.availableMB‚ 1943.71875 disk.sda2.out‚0 disk.dm-1.in‚ 5.93359375& disk.nvme0c0n1.out‚68377.224609375 proc.cpu.threads‚17 network.recv‚ 2.215231e+06 disk.sda1.out‚0 disk.sda.out‚ 467.4765625 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0 disk.dm-0.in‚ 18317.265625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 467.4765625 disk.sda3.out‚ 467.4765625# proc.memory.rssMB‚ 2943.29296875) disk./.usagePercent‚39.04345611154294 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19353.40380859375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625! disk.dm-0.out‚68377.224609375$ memory_percent‚88.13648223876953 network.sent‚ 9.770633e+06 cpu‚25.587604492851465 disk.sda2.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚581.7757110595703 disk.rbd2.out‚58.78564453125" disk.nvme0c0n1.in‚ 18317.265625) proc.memory.percent‚17.96443462371826ãTõ2µ- _wandbruntime‚8632  _runtime‚8632šCǶÁ & lr-AdamW/pg1‚0.0002354861974401991 & lr-AdamW/pg2‚0.0002354861974401991 trainer/global_step‚5199 % _timestamp‚1.7843283789579434e+09 _runtime‚8632.332565781 _step‚424Ùq&òÌ·Æ & lr-AdamW/pg1‚0.0002354861974401991 & lr-AdamW/pg2‚0.0002354861974401991 trainer/global_step‚5199 % _timestamp‚1.7843283789579434e+09 _runtime‚8632.332565781 _step‚424¨a Ü‹2¸- _wandbruntime‚8632  _runtime‚8632LÆxVÿ'¹ùO trainer/global_step‚5199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28081527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44982307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44259643 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44259637 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4887608652800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843283789590247e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6304428030976 _step‚425 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4879248720325 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1491627076042752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251923 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1496514684695552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1496514684695552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1491627049869312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28081275 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1496513313132544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410003248916480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403698418257920 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89241944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13334820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6304462567424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4877904120003 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1491627049869312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495861916258757 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495860546230467 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _runtime‚8632.337353218 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14746707 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14746460 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490982667538432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44982307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44992280 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1491627076042752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4886263263232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44259643 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44992280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410002846288896 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1490982642110464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4886263263232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496513313132544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887608652800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13334815 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403698786349056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820mXú6ɺþO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89241944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13334820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6304462567424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4877904120003 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1491627049869312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495861916258757 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495860546230467 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _runtime‚8632.337353218 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14746707 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14746460 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490982667538432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44982307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44992280 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1491627076042752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4886263263232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44259643 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44992280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410002846288896 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1490982642110464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4886263263232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496513313132544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887608652800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13334815 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403698786349056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚5199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28081527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44982307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44259643 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44259637 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ]ODÍŸÇ¡;eviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4887608652800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843283789590247e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6304428030976 _step‚425 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4879248720325 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1491627076042752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251923 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1496514684695552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1496514684695552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1491627049869312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28081275 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1496513313132544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410003248916480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403698418257920 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995©ÅÜ%Ó2»- _wandbruntime‚8633  _runtime‚8633¥^û '¼ƒN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1491905076772864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888533695488 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268621 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1491905050599424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887188298752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13337358 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1491905050599424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44267849 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28086800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403775904665600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚8633.731000908 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44267843 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28086559 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4878827586249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44267843 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491260523888640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14749437 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410081524093952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚5199 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491260549316608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6305653957632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403776272756736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1491905076772864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888533695488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6305619428352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KgŠ'½ˆN G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268621 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89258628 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44267849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 % _timestamp‚1.7843283803561478e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45000772 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44990785 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496793610468352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13337363 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45000772 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1496793610468352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268621 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1491905050599424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887188298752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13337358 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1491905050599424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44267849 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28086800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403775904665600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚8633.731000908 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44267843 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28086559 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4878827586249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44267843 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491260523888640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14749437 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410081524093952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚5199 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491260549316608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6305653957632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403776272756736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1491905076772864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888533695488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6305619428352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1491905076772864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888533695488 E&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992207944393158 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28398022055625916 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7778528332710266 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5763924717903137 +jet/accuracy_2_of_2‚0.13781693577766418 6validation_average_jet_accuracy‚0.3427993357181549 trainer/global_step‚5199 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9881633520126343 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692571520805359 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847847819328308 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03898533061146736 *jet/accuracy_1_of_3‚0.GY‘xÜ,9057917594909668 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47350820899009705 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +particle/sensitivity‚0.5945603251457214 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6443814635276794 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.614932656288147 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959149956703186 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.634444534778595 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05713922530412674 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12423370778560638 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15666666626930237 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8157188892364502 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14025869965553284 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8254913091659546 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7188711762428284 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05713922530412674 +jet/accuracy_3_of_4‚0.08617176860570908 /particle/accuracy_3_of_4‚0.9965986609458923 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17437942326068878 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.612411379814148 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15571023523807526 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17756414413452148 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6994624733924866 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13037832081317902 particle/accuracy_2_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6135903596878052 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8613831400871277 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +jet/accuracy_3_of_3‚0.07163582742214203 *jet/accuracy_2_of_4‚0.7574212551116943 _step‚428 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16967085003852844 particle/accuracy_3_of_3‚1 jet/accuracy_5_of_5‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14075729250907898 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9307473301887512 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6854462623596191 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6048984527587891 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.813246488571167 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9829505681991577 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6847396492958069 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6394607424736023 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18247753381729126 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_3‚0.5288459658622742 jet/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1646861732006073 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.667492687702179 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7146058082580566 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7367731928825378 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 #particle/accuracy‚ 0.84423828125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6952088475227356 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6546101570129395 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906876683235168 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6861404180526733 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906860589981079 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7474983334541321 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562499761581421 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.671875 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.655223548412323 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.609415590763092 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6689944863319397 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027385802939534187 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17122776806354523 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.609415590763092 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8191989064216614 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6927204132080078 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24549509584903717 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5518708825111389 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3905889391899109 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03166734427213669 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18606553971767426 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967160224914551 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4326646029949188 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8157188892364502 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ,"Purity/2FRt0SRqqt1FBt/event_purity‚0.375 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16041480004787445 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18942579627037048 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9125170111656189 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18560533225536346 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6668244004249573 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6640196442604065 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17150862514972687 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 _runtime‚8660.138513694 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17907460033893585 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007260125130414963 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16647450625896454 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 )loss/val_total_loss‚8.476535364616822 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8155329823493958 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6469603180885315 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44054603576660156 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05602993816137314 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.919446587562561 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99530029296875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21844971179962158 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.993653416633606 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7285718321800232 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.055616237223148346 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20113295316696167 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.053627222776412964 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013252448290586472 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822498559951782 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7456727623939514 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01685400865972042 *jet/accuracy_1_of_2‚0.7587089538574219 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9863999485969543 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943559765815735 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.054207392036914825 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962416887283325 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5877121686935425 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6576439142227173 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46506527066230774 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7357178926467896 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17150862514972687 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7137832045555115 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843284067632349e+09†€â=ÇÜ{ 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14025869965553284 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8254913091659546 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7188711762428284 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05713922530412674 +jet/accuracy_3_of_4‚0.08617176860570908 /particle/accuracy_3_of_4‚0.9965986609458923 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17437942326068878 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.612411379814148 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15571023523807526 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17756414413452148 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6994624733924866 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13037832081317902 particle/accuracy_2_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6135903596878052 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8613831400871277 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +jet/accuracy_3_of_3‚0.07163582742214203 *jet/accuracy_2_of_4‚0.7574212551116943 _step‚428 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16967085003852844 particle/accuracy_3_of_3‚1 jet/accuracy_5_of_5‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14075729250907898 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9307473301887512 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6854462623596191 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6048984527587891 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.813246488571167 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9829505681991577 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6847396492958069 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6394607424736023 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18247753381729126 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_3‚0.5288459658622742 jet/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1646861732006073 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.667492687702179 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7146058082580566 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7367731928825378 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 #particle/accuracy‚ 0.84423828125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6952088475227356 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6546101570129395 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906876683235168 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6861404180526733 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906860589981079 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7474983334541321 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562499761581421 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.671875 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.655223548412323 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.609415590763092 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6689944863319397 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027385802939534187 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17122776806354523 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.609415590763092 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8191989064216614 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6927204132080078 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24549509584903717 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5518708825111389 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3905889391899109 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03166734427213669 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18606553971767426 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967160224914551 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4326646029949188 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8157188892364502 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ,"Purity/2FRt0SRqqt1FBt/event_purity‚0.375 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16041480004787445 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18942579627037048 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9125170111656189 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18560533225536346 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6668244004249573 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6640196442604065 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17150862514972687 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 _runtime‚8660.138513694 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17907460033893585 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007260125130414963 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16647450625896454 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 )loss/val_total_loss‚8.476535364616822 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8155329823493958 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6469603180885315 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44054603576660156 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05602993816137314 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.919446587562561 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99530029296875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21844971179962158 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.993653416633606 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7285718321800232 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.055616237223148346 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20113295316696167 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.053627222776412964 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013252448290586472 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822498559951782 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7456727623939514 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01685400865972042 *jet/accuracy_1_of_2‚0.7587089538574219 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9863999485969543 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943559765815735 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.054207392036914825 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962416887283325 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5877121686935425 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6576439142227173 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46506527066230774 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7357178926467896 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17150862514972687 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7137832045555115 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843284067632349e+09 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23802998661994934 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6906206011772156 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9763268828392029 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6988182067871094 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.531374454498291 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5808690190315247 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 epoch‚12 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35699114203453064 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4017016589641571 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 *jet/accuracy_1_of_1‚0.2968972325325012 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6211182475090027 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17071422934532166 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 /particle/accuracy_4_of_4‚0.9120740294456482 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15622392296791077 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36037927865982056 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_4_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29906904697418213 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8143990635871887 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15153487026691437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1350780576467514 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9689728617668152 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6289326548576355 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8762867450714111 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4027777910232544 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992207944393158 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28398022055625916 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7778528332710266 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5763924717903137 +jet/accuracy_2_of_2‚0.13781693577766418 6validation_average_jet_accuracy‚0.3427993357181549 trainer/global_step‚5199 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9881633520126343 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692571520805359 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847847819328308 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03898533061146736 *jet/accuracy_1_of_3‚0.9057917594909668 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47350820899009705 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +particle/sensitivity‚0.5945603251457214 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6443814635276794 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.614932656288147 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959149956703186 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.634444534778595 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05713922530412674 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12423370778560638 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15666666626930237 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8157188892364502 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141¬£¼LžÈ‚(:“ àáêÒ¼˜×± gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UŠF:É‚(:¯ àáêÒÓŠˆ´ disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125# disk.dm-0.out‚70159.21630859375( disk.nvme0c0n1.out‚70159.21630859375 disk.sda.in‚ 5.93359375 disk.sda.out‚ 470.23828125 disk.rbd1.in‚ 506.52734375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 9.843418e+06 network.recv‚ 2.227047e+06) disk./.usagePercent‚38.98984045364699" disk.nvme0c0n1.in‚ 18466.421875 disk.nvme0n1.in‚ 18466.421875 disk.dm-0.in‚ 18466.421875! proc.memory.rssMB‚ 2950.078125 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 470.23828125$ disk./.usageGB‚580.9767990112305 disk.rbd2.in‚0.015625 disk.sda1.in‚0* proc.memory.percent‚18.005847930908203& disk.nvme0n1.out‚70159.21630859375 disk.sda2.in‚0 disk.drbd2165.out‚0 cpu‚25.592777337542437 disk.rbd0.in‚0 disk.drbd2165.in‚0$ memory_percent‚82.18550682067871( proc.memory.availableMB‚ 2918.7265625 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 470.23828125 disk.sda2.out‚0# disk.rbd1.out‚19353.60693359375Ó攔ʂ(:‰ ïáêÒһ⬠gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áª¦Ð:Ë‚(:¯ ïáêÒÂɾ¯% disk.nvme0n1.out‚70168.4013671875) proc.memory.availableMB‚ 2918.05859375) disk./.usagePercent‚38.98974342679151 disk.dm-0.in‚ 18467.3828125 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.rbd1.in‚ 506.52734375! disk.nvme0n1.in‚ 18467.3828125 cpu‚25.59241601603457 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 471.10546875* proc.memory.percent‚18.005847930908203 proc.cpu.threads‚17 network.sent‚ 9.901286e+06 disk.sda.out‚ 471.10546875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 471.10546875' disk.nvme0c0n1.out‚70168.4013671875! proc.memory.rssMB‚ 2950.078125# disk.nvme0c0n1.in‚ 18467.3828125$ disk./.usageGB‚580.9753532409668 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125" disk.dm-0.out‚70168.4013671875 disk.sda3.in‚ 5.93359375 network.recv‚ 2.232352e+06 disk.sda2.in‚0# disk.rbd1.out‚19353.60693359375$ memory_percent‚82.18958377838135 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0üû£ŸÌ‚(:” þáêҊë¬ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü?Í‚(:´ þáêÒþéç®& disk.nvme0n1.out‚71148.82958984375 disk.rbd1.in‚ 506.52734375" disk.nvme0n1.in‚18468.21484375$ disk./.usageGB‚580.9718589782715 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda3.in‚ 5.93359375) disk./.usagePercent‚38.98950892389015 disk.drbd2165.in‚0 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2950.078125 cpu‚25.592060662872125 disk.sda.in‚ 5.93359375 network.recv‚ 2.235818e+06 disk.sda2.in‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚18468.21484375# disk.dm-0.out‚71148.82958984375 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 network.sent‚ 9.906363e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 471.41796875$ disk.nvme0c0n1.in‚18468.21484375 disk.sda.out‚ 471.41796875$ memory_percent‚87.84551620483398' proc.memory.availableMB‚ 1991.390625 disk.sda2.out‚0# disk.rbd1.out‚19353.68505859375( disk.nvme0c0n1.out‚71148.82958984375* proc.memory.percent‚18.005847930908203 proc.cpu.threads‚17 disk.sda3.out‚ 471.41796875‘>1)žÎ‚(:“ âêÒñ²ß­ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±PÑåÚÏ‚(:£ âêÒòØö®! disk.nvme0n1.out‚ 71152.203125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 71152.203125" disk.nvme0n1.in‚18655.98046875 disk.sda.out‚ 471.7265625$ memory_percent‚88.12499046325684 disk.dm-1.out‚ 471.7265625 disk.sda2.in‚0# disk.rbd1.out‚19353.68505859375 disk.rbd0.in‚0 disk.dm-0.in‚18655.98046875 disk.dm-0.out‚ 71152.203125 disk.drbd2165.in‚0 disk.sda.ilp‘Tn‚ 5.93359375 disk.sda1.out‚0* proc.memory.percent‚18.005847930908203) disk./.usagePercent‚38.98937938407782 disk.sda3.in‚ 5.93359375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1945.6015625$ disk./.usageGB‚580.9699287414551$ disk.nvme0c0n1.in‚18655.98046875 disk.rbd1.in‚ 506.52734375 proc.cpu.threads‚17 network.sent‚ 9.911474e+06 disk.dm-1.in‚ 5.93359375 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125! proc.memory.rssMB‚ 2950.078125 cpu‚25.591730331395325 network.recv‚ 2.239288e+06 disk.sda3.out‚ 471.7265625ÃzÁRŸÐ‚(:” œâêÒÔç¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚217.128' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.425599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-àßÉ3Ñ‚(:¨ œâêÒÁⲯ disk.rbd1.in‚ 506.52734375 disk.dm-1.in‚ 5.93359375# disk.dm-0.out‚71161.08349609375* proc.memory.percent‚18.005847930908203 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 18656.9375 disk.sda3.in‚ 5.93359375! proc.memory.rssMB‚ 2950.078125 network.recv‚ 2.243011e+06 disk.nvme0c0n1.in‚ 18656.9375 disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0$ disk./.usageGB‚581.4198875427246 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19353.81005859375 proc.cpu.threads‚17 network.sent‚ 9.916562e+06 disk.nvme0n1.in‚ 18656.9375& disk.nvme0n1.out‚71161.08349609375 disk.sda2.in‚0) proc.memory.availableMB‚ 1945.97265625 cpu‚25.59131333129287( disk.nvme0c0n1.out‚71161.08349609375 disk.rbd0.out‚0 disk.dm-1.out‚ 472.18359375$ memory_percent‚88.12272548675537 disk.sda3.out‚ 472.18359375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk./.usagePercent‚39.0195764967716 disk.sda.out‚ 472.18359375üNÔ:2Ò- _wandbruntime‚8742  _runtime‚8742©5ÈÓ ' lr-AdamW/pg1‚0.00023471915738229525 ' lr-AdamW/pg2‚0.00023471915738229525 trainer/global_step‚5249 % _timestamp‚1.7843284891462398e+09 _runtime‚ 8742.52078246 _step‚429&¼ù4ÍÔÇ trainer/global_step‚5249 % _timestamp‚1.7843284891462398e+09 _runtime‚ 8742.52078246 _step‚429 ' lr-AdamW/pg1‚0.00023471915738229525 ' lr-AdamW/pg2‚0.00023471915738229525­¾ 2Õ- _wandbruntime‚8742  _runtime‚8742µíöaÿ'ÖùO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4938421808128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413992780558848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45477019 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚8742.521992845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 % _timestamp‚1.7843284891472769e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4929970364553 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268989 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45477019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514294861180928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1508702665379840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4938421808128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6371962109440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514294861180928 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1509355093983232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45467046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939767197696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4939767197696 _step‚430 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13470888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44801943 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407620850888704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28389132 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14918249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28389374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407621218979840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1509355093983232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6371929670144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931314964875 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514293489617920 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1509355067809792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14918486 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4929970364553 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268989 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45477019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514294861180928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1508702665379840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4938421808128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6371962109440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514294861180928 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1509355093983232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45467046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939767197696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4939767197696 _step‚430 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13470888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44801943 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407620850888704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28389132 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14918249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28389374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407621218979840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1509355093983232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6371929670144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931314964875 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514293489617920 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1509355067809792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14918486 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1509633094713344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13473426 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28394451 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1513912785589249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45475524 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6373177108992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1508980547158016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 % _timestamp‚1.7843284905312462e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28394684 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44810149 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285673 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚431 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44810155 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285673 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295666 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1509633068539904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4940692240384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13473431 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295666 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407698337296384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1513911415560847 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414071882496512 _runtime‚8743.906107734 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407698705387520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4932238431233 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1509633094713344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1509633068539904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45485511 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14921253 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1509633068539904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4940692240384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13473431 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295666 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407698337296384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1513911415560847 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414071882496512 _runtime‚8743.906107734 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407698705387520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4932238431233 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1509633094713344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1509633068539904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45485511 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14921253 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1509633094713344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13473426 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28394451 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1513912785589249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45475524 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6373177108992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1508980547158016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 % _timestamp‚1.7843284905312462e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28394684 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44810149 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285673 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚431 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44810155 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285673 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295666¯! 2Û- _wandbruntime‚8743  _runtime‚8743ÞÄé ܃ 0loss/FRt1/assignment_loss‚0.7954225847080573 /loss/FBt1/detection_loss‚0.4790283186966858 2loss/SRqqt2/assignment_loss‚0.6305748396603577 /loss/FRt1/detection_loss‚1.5395763316933084 /loss/FRt2/detection_loss‚0.5928746006724133 1loss/SRqqt1/detection_loss‚0.6135007239204109 1loss/SRqqt2/detection_loss‚0.9176179743837565 %loss/total_loss‚8.808557802569261 epoch‚13 0loss/FRt2/assignment_loss‚1.8073444697656669 2loss/SRqqt1/assignment_loss‚0.5118218227665003 /loss/FBt2/detection_loss‚0.9048099733228164 trainer/global_step‚5249 % _timestamp‚1.7843284906176064e+09 _runtime‚ 8743.99199996 _step‚432 1loss/FBt1/assignment_loss‚0.01596179589002194 3loss/FBt2/assignment_loss‚2.436708926447937e-05e¶aâ݈ 2loss/SRqqt2/assignment_loss‚0.6305748396603577 /loss/FRt1/detection_loss‚1.5395763316933084 /loss/FRt2/detection_loss‚0.5928746006724133 1loss/SRqqt1/detection_loss‚0.6135007239204109 1loss/SRqqt2/detection_loss‚0.9176179743837565 %loss/total_loss‚8.808557802569261 epoch‚13 0loss/FRt2/assignment_loss‚1.8073444697656669 2loss/SRqqt1/assignment_loss‚0.5118218227665003 /loss/FBt2/detection_loss‚0.9048099733228164 trainer/global_step‚5249 % _timestamp‚1.7843284906176064e+09 _runtime‚ 8743.99199996 _step‚432 1loss/FBt1/assignment_loss‚0.01596179589002194 3loss/FBt2/assignment_loss‚2.436708926447937e-05 0loss/FRt1/assignment_loss‚0.7954225847080573 /loss/FBt1/detection_loss‚0.4790283186966858°Œ­aŸÞ‚(:” «âêÒ‹ºˆ¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žg²Aß‚(:¶ «âêÒÊ“ì­" disk.nvme0n1.in‚18657.76953125 disk.dm-0.in‚18657.76953125* proc.memory.percent‚18.005847930908203 disk.rbd0.in‚0) proc.memory.availableMB‚ 1945.44140625 network.recv‚ 2.246461e+06 disk.sda2.out‚0# disk.dm-0.out‚71750.94677734375 disk.rbd1.in‚ 506.52734375 disk.dm-1.in‚ 5.93359375! proc.memory.rssMB‚ 2950.078125 proc.cpu.threads‚17) disk./.usagePercent‚39.04101815179437 disk.sda3.out‚ 472.53515625 disk.sda2.in‚0 disk.rbd2.out‚58.78564453125$ memory_percent‚88.12596797943115$ disk./.usageGB‚581.7393836975098& disk.nvme0n1.out‚71761.07177734375# disk.rbd1.out‚19353.81005859375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375 disk.sda.out‚ 472.53515625 disk.dm-1.out‚ 472.53515625 cpu‚25.591024056206468$ disk.nvme0c0n1.in‚18657.76953125( disk.nvme0c0n1.out‚71761.07177734375 disk.sda.in‚ 5.93359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 9.921671e+06ür”à‚(:‰ ºâêÒ•äЫ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚306.464' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.2928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04µèð>á‚(:³ ºâêÒ±öØ®" disk.dm-0.out‚71986.5869140625 disk.sda1.out‚0) proc.memory.availableMB‚ 1945.55078125$ disk./.usageGB‚581.7127838134766% disk.nvme0n1.out‚71986.5869140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375$ memory_percent‚88.12530040740967 proc.cpu.threads‚17 disk.sda3.out‚ 472.72265625 disk.dm-1.out‚ 472.72265625 disk.sda.out‚ 472.72265625 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚18657.76953125 disk.dm-1.in‚ 5.93359375* disk./.usagePercent‚39.039233011258105# disk.rbd1.out‚19353.88818359375 disk.sda1.in‚0* proc.memory.percent‚18.005847930908203 cpu‚25.59066423729364' disk.nvme0c0n1.out‚71986.5869140625 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚18657.76953125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375! proc.memory.rssMB‚ 2950.078125 disk.drbd2165.in‚0 network.sent‚ 9.985054e+06 network.recv‚ 2.251033e+06" disk.nvme0n1.in‚18657.76953125r=§òžâ‚(:“ Éâê҉虭 gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¨•¶;ã‚(:° ÉâêÒ殓¯) disk./.usagePercent‚38.98951839616892 disk.sda1.out‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1945.55078125! proc.memory.rssMB‚ 2950.078125& disk.nvme0n1.out‚71994.47607421875 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.out‚ 473.0703125 cpu‚25.590279632963686$ disk./.usageGB‚580.9720001220703 disk.sda.out‚ 473.0703125 network.sent‚ 9.989659e+06 disk.sda3.out‚ 473.0703125# disk.rbd1.out‚19353.88818359375 disk.dm-0.in‚ 18831.1328125$ memory_percent‚88.12530040740967 disk.rbd1.in‚ 506.52734375# disk.dm-0.out‚71994.47607421875 disk.dm-1.in‚ 5.93359375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚71994.47607421875! disk.nvme0n1.in‚ 18831.1328125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 2.254628e+06 disk.sda3.in‚ 5.93359375# disk.nvme0c0n1.in‚ 18831.1328125 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125 disk.sda2.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.005847930908203©e/”ä‚(:‰ ØâêÒ‰•‰¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚343.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÀ>>å‚(:³ ØâêÒç§Þ® disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19353.95068359375) proc.memory.availableMB‚ 1945.30078125 disk.sda1.in‚0 disk.dm-1.out‚ 473.2578125 disk.sda3.in‚ 5.93359375" disk.nvme0n1.in‚18831.96484375 disk.sda.out‚ 473.2578125! proc.memory.rssMB‚ 2950.078125 disk.dm-1.in‚ 5.93359375) disk./.usagePercent‚38.98770458278882 disk.rbd2.out‚58.78564453125# disk.dm-0.out‚72005.37255859375 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚72005.37255859375$ disk./.usageGB‚580.9449729919434 disk.sda3.out‚ 473.2578125 disk.sda2.in‚0 cpu‚25.589986816339348 network.sent‚ 9.994346e+06 network.recv‚ 2.258102e+06$ disk.nvme0c0n1.in‚18831.96484375( disk.nvme0c0n1.out‚72005.37255859375 disk.rbd0.out‚0* proc.memory.percent‚18.005847930908203 disk.dm-0.in‚18831.96484375$ memory_percent‚88.12682628631592 proc.cpu.threads‚17 disk.sda.in‚ 5.93359375S‹V(Ÿæ‚(:” çâêÒ‘»²­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)l |-ç‚(:¢ çâêÒûÐÏ® disk.sda1.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚18831.96484375) proc.memory.availableMB‚ 1947.01953125! proc.memory.rssMB‚ 2950.078125 disk.sda.in‚ 5.93359375 disk.sda2.in‚0# disk.rbd1.out‚19353.95068359375 disk.rbd2.in‚0.015625$ memory_percent‚88.11633586883545 disk.sda1.out‚0 disk.sda3.out‚473.5625 disk.sda.out‚473.5625 proc.cpu.threads‚17 disk.dm-1.out‚473.5625& disk.nvme0c0n1.out‚72009.759765625" disk.nvme0n1.in‚18831.96484375 disk.rbd2.out‚58.78564453125! disk.dm-0.out‚72009.759765625 cpu‚25.589608051487513 network.recv‚ 2.26188e+06 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚18831.96484375$ disk./.usageGB‚580.9452285766602$ disk.nvme0n1.out‚72009.759765625 disk.sda2.out‚0 network.sent‚ 9.99942e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚18.005847930908203) disk./.usagePercent‚38.98772173529362Ojì”è‚(:‰ öâêÒ—Àú« gpu.0.gpu‚68.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚338.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XUqÊ6é‚(:« öâêÒ”¡ö­ disk.sda1.out‚0" disk.nvme0c0n1.in‚ 18832.921875 disk.sda2.in‚0$ memory_percent‚88.11779022216797 proc.cpu.threads‚17# disk.dm-0.out‚72021.88330078125 disk.sda1.in‚0# disk.rbd1.out‚19354.02880859375* proc.memory.percent‚18.005847930908203 cpu‚25.589290743228425 disk.sda.in‚ 5.93359375 disk.rbd2.out‚58.78564453125 network.sent‚ 1.0004528e+07 disk.dm-0.in‚ 18832.921875 disk.sda3.out‚ 473.7734375 disk.nvme0n1.in‚ 18832.921875 disk.sda.out‚ 473.7734375 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚38.98771200700731$ disk./.usageGB‚580.9450836181641& proc.memory.availableMB‚ 1946.78125! proc.memory.rssMB‚ 2950.078125 disk.dm-1.out‚ 473.7734375( disk.nvme0c0n1.out‚72021.88330078125& disk.nvme0n1.out‚72021.88330078125 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375 network.recv‚ 2.265098e+06 disk.drbd2165.out‚0 disk.sda2.out‚0i ƒ¨2ê- _wandbruntime‚8821  _runtime‚8821Aëa1Éëà _runtime‚8821.367448446 _step‚433 ' lr-AdamW/pg1‚0.00023394584313819945 ' lr-AdamW/pg2‚0.00023394584313819945 trainer/global_step‚5299 % _timestamp‚1.7843285679917943e+09°+%íÎìÈ ' lr-AdamW/pg1‚0.00023394584313819945 ' lr-AdamW/pg2‚0.00023394584313819945 trainer/global_step‚5299 % _timestamp‚1.7843285679917943e+09 _runtime‚8821.367448446 _step‚433±’aé²2í- _wandbruntime‚8821  _runtime‚8821(-Ýþ'îøO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚411495171272704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1528241213617152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15056528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417927110369792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45902771 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15056282 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843285679926155e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28654566 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1522596754288640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45892798 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4976207477661 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1522596728860672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45212249 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45212249 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4984737738752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45212243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚434 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1523255104315392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411495539363840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1528239842054144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523255130488832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6431939097088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4984737738752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28654315 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45902771 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45212243 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45892798 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115020 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1523255130488832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13598033 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4977552077983 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4986083128320 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175NrïýO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1522596728860672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45212249 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45212249 NEDeviceStatsMonitor.on_train_bat‘<4‘$ch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4984737738752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45212243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚434 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1523255104315392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411495539363840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1528239842054144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523255130488832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6431939097088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4984737738752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28654315 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45902771 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45212243 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45892798 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115020 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1523255130488832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13598033 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4977552077983 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4986083128320 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚411495171272704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1528241213617152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15056528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417927110369792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45902771 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15056282 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843285679926155e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28654566 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1522596754288640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45892798 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4976207477661 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640²Uºb2ð- _wandbruntime‚8822  _runtime‚8822³Î) 'ñƒN UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4985662774272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411573025771520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45220455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4977130943907 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4985662774272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1527853111611157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45911263 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45220455 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45911263 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45901276 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45901276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28659823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418006195890688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411572657680384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚8822.768881545 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚435 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131718 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28659580 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131718 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418005793270272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978475544341 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1528518767819776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1528520139389952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13600581 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121725 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523533131218944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15059004 trainer/global_step‚5299 % _timestamp‚1.7843285693939893e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1527851741582755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1522874610638848 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987008171008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13600576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0b©™'òˆN P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1528520139389952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13600581 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121725 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523533131218944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15059004 trainer/global_step‚5299 % _timestamp‚1.7843285693939893e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1527851741582755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1522874610638848 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987008171008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13600576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4985662774272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411573025771520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45220455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4977130943907 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4985662774272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1527853111611157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45911263 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45220455 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45911263 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45901276 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45901276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28659823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418006195890688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411572657680384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚8822.768881545 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚435 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131718 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28659580 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131718 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418005793270272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978475544341 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1528518767819776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0³i"#(2ó- _wandbruntime‚8822  _runtime‚8822é£< ôƒ 2loss/SRqqt1/assignment_loss‚0.5351925530171027 /loss/FRt1/detection_loss‚1.3989107642519067 /loss/FBt2/detection_loss‚0.6077658750436967 _runtime‚8822.848786275 2loss/SRqqt2/assignment_loss‚0.4913629222648245 1loss/FBt1/assignment_loss‚0.01081097685606593 2loss/FBt2/assignment_loss‚0.005127693635571973 /loss/FRt2/detection_loss‚0.5798041980532016 0loss/FRt1/assignment_loss‚0.7765885114402514 0loss/SRqqt1/detection_loss‚0.650686082347407 0loss/FBt1/detection_loss‚0.40728428397716204 %loss/total_loss‚8.058205495448728 epoch‚13 trainer/global_step‚5299 _step‚436 0loss/FRt2/assignment_loss‚1.6173299548065239 1loss/SRqqt2/detection_loss‚0.9773416797550151 % _timestamp‚1.7843285694741995e+09¥úV¿õˆ trainer/global_step‚5299 _step‚436 0loss/FRt2/assignment_loss‚1.6173299548065239 1loss/SRqqt2/detection_loss‚0.9773416797550151 % _timestamp‚1.7843285694741995e+09 2loss/SRqqt1/assignment_loss‚0.5351925530171027 /loss/FRt1/detection_loss‚1.3989107642519067 /loss/FBt2/detection_loss‚0.6077658750436967 _runtime‚8822.848786275 2loss/SRqqt2/assignment_loss‚0.4913629222648245 1loss/FBt1/assignment_loss‚0.01081097685606593 2loss/FBt2/assignment_loss‚0.005127693635571973 /loss/FRt2/detection_loss‚0.5798041980532016 0loss/FRt1/assignment_loss‚0.7765885114402514 0loss/SRqqt1/detection_loss‚0.650686082347407 0loss/FBt1/detection_loss‚0.40728428397716204 %loss/total_loss‚8.058205495448728 epoch‚13´¼ ¤”ö‚(:‰ …ãêÒ¦¬ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹89÷‚(:® …ãêÒÉäê® network.sent‚ 1.0010124e+07$ disk./.usageGB‚580.9451293945312 disk.sda3.in‚ 5.93359375 disk.sda2.in‚0! proc.memory.rssMB‚ 2950.078125* proc.memory.percent‚18.005847930908203 disk.rbd1.in‚ 506.52734375 disk.sda3.out‚ 474.01953125( proc.memory.availableMB‚ 1946.4453125# disk.rbd1.out‚19354.02880859375 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚72025.28515625$ memory_percent‚88.11984062194824 cpu‚25.588936532821815 network.recv‚ 2.268421e+06 disk.dm-0.in‚18873.69921875 disk.drbd2165.out‚0 disk.dm-1.out‚ 474.01953125" disk.nvme0n1.in‚18873.69921875 disk.sda.out‚ 474.01953125* disk./.usagePercent‚38.987715079097725 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚18873.69921875# disk.nvme0n1.out‚72025.28515625 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.out‚72025.28515625GøÅÊø‚(:‰ ”ãêÒ…ýº® gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚26Qg,Ê7.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x J<ù‚(:± ”ãêÒà‹Í° disk.sda.out‚ 474.3203125 disk.dm-0.in‚18984.59765625 disk.sda3.out‚ 474.3203125$ disk.nvme0c0n1.in‚18984.59765625 proc.cpu.threads‚17# disk.rbd1.out‚19354.09130859375& disk.nvme0n1.out‚72035.23876953125' proc.memory.availableMB‚ 1946.421875 network.recv‚ 2.273269e+06) disk./.usagePercent‚38.98753177770321$ disk./.usageGB‚580.9423980712891 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 474.3203125 cpu‚25.588578428742206 network.sent‚ 1.0072633e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0# memory_percent‚88.1199836730957 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0# disk.dm-0.out‚72035.23876953125 disk.dm-1.in‚ 5.93359375( disk.nvme0c0n1.out‚72035.23876953125* proc.memory.percent‚18.005847930908203 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda3.in‚ 5.93359375" disk.nvme0n1.in‚18984.59765625! proc.memory.rssMB‚ 2950.078125%Ümß”ú‚(:‰ £ãêÒ»¦ä¬ gpu.0.gpu‚48.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚156.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2428 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Ñ+ƒ:û‚(:¯ £ãêÒÑÞ¯ network.sent‚ 1.0077796e+07 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 network.recv‚ 2.27683e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375! proc.memory.rssMB‚ 2950.078125 disk.sda1.in‚0 disk.dm-1.out‚ 474.63671875 disk.sda3.out‚ 474.63671875& disk.nvme0n1.out‚72043.03271484375 disk.sda.in‚ 5.93359375$ disk./.usageGB‚580.9452705383301 disk.sda2.in‚0 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 18985.5546875# disk.dm-0.out‚72043.03271484375# disk.nvme0c0n1.in‚ 18985.5546875( disk.nvme0c0n1.out‚72043.03271484375$ proc.memory.availableMB‚1946.875* disk./.usagePercent‚38.987724551376495 disk.rbd2.out‚58.78564453125$ memory_percent‚88.11721801757812* proc.memory.percent‚18.005847930908203 cpu‚25.588296486794125 proc.cpu.threads‚17# disk.rbd1.out‚19354.09130859375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 18985.5546875 disk.sda.out‚ 474.63671875±ñ:@Ÿü‚(:” ²ãêÒ×™ë¬ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.444799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qnå%>ý‚(:³ ²ãêÒÆÙ—¯ disk.drbd2165.out‚0$ disk./.usageGB‚580.9451789855957 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚25.587923469166917 disk.rbd0.out‚0& disk.nvme0n1.out‚72048.55419921875 disk.sda2.out‚0! proc.memory.rssMB‚ 2950.078125 disk.drbd2165.in‚0) disk./.usagePercent‚38.98771840719567 disk.sda3.out‚ 474.80859375$ memory_percent‚88.11745643615723 disk.dm-1.out‚ 474.80859375 proc.cpu.threads‚17 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0( disk.nvme0c0n1.out‚72048.55419921875 disk.rbd1.in‚ 506.52734375 network.recv‚ 2.279967e+06# disk.nvme0c0n1.in‚ 18985.5546875! disk.nvme0n1.in‚ 18985.5546875 disk.sda.in‚ 5.93359375 disk.sda.out‚ 474.80859375 disk.rbd2.in‚0.015625# disk.dm-0.out‚72048.55419921875 disk.sda1.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375# disk.rbd1.out‚19354.23193359375( proc.memory.availableMB‚ 1946.8359375 network.sent‚ 1.0082902e+07 disk.dm-0.in‚ 18985.5546875* proc.memory.percent‚18.005847930908203¿¤Û”þ‚(:‰ ÁãêÒ…¹Ô« gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨®"<ÿ‚(:± ÁãêÒ¤„š® disk.drbd2165.out‚0 disk.sda3.out‚ 475.09765625% disk.nvme0n1.out‚72056.8681640625 disk.sda.in‚ 5.93359375 disk.rbd0.out‚0$ disk./.usageGB‚580.9452171325684" disk.dm-0.out‚72056.8681640625 disk.sda3.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375 cpu‚25.587585324506 network.sent‚ 1.0087964e+07) disk./.usagePercent‚38.98772096727101 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125* proc.memory.percent‚18.005847930908203 disk.sda2.out‚0# disk.rbd1.out‚19354.23193359375 disk.dm-0.in‚18986.38671875 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375' disk.nvme0c0n1.out‚72056.8681640625$ memory_percent‚88.11757564544678 proc.cpu.threads‚17 network.recv‚ 2.283434e+06 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚18986.38671875 disk.sda2.in‚0) proc.memory.availableMB‚ 1946.81640625 disk.rbd0.in‚0 disk.dm-1.out‚ 475.09765625" disk.nvme0n1.in‚18986.38671875 disk.sda.out‚ 475.09765625! proc.memory.rssMB‚ 2950.078125!Û4o2€- _wandbruntime‚8901  _runtime‚8901Ù+ÅÈ ' lr-AdamW/pg2‚0.00023316630437644054 trainer/global_step‚5349 $ _timestamp‚1.784328648326553e+09 _runtime‚8901.701164575 _step‚437 ' lr-AdamW/pg1‚0.00023316630437644054ÌžÍ‚Ç _step‚437 ' lr-AdamW/pg1‚0.00023316630437644054 ' lr-AdamW/pg2‚0.00023316630437644054 trainer/global_step‚5349 $ _timestamp‚1.784328648326553e+09 _runtime‚8901.701164575µU¬­%2ƒ- _wandbruntime‚8901  _runtime‚8901§î7ÿ'„ùO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45622543 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1542186194490368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415369491656704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1537155166994432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46318550 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421861651085824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5031053669376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1542186194490368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15193426 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6491756801536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941093 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13725183 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5023789191091 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚5349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5031053669376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536490843197440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5032399058944 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1542187566053376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1537155166994432 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1537155140820992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46328523 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45622543 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6491791337984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5022444590769 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415369859747840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843286483277209e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5032399058944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421861248458240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1537155140820992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872â~T¹(…þO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46328523 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45622543 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6491791337984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5022444590769 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415369859747840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843286483277209e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5032399058944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421861248458240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1537155140820992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45622543 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1542186194490368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415369491656704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1537155166994432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46318550 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421861651085824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5031053669376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1542186194490368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15193426 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6491756801536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941093 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13725183 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5023789191091 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚5349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5031053669376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536490843197440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5032399058944 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1542187566053376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1537155166994432 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1537155140820992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6¶i"ÆÞ2†- _wandbruntime‚8903  _runtime‚8903ħᇂN F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033324101632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚5349 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1537433141551104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46327028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15196122 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421940329414144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6492983258624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45630749 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541793437633065 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1537433167724544 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415446978064384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6492948729344 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967770 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15196361 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 $ _timestamp‚1.784328649722187e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1542466491826176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1542466491826176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46327028 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957777 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28923848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024712657449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46337015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033324101632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚5349 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1537433141551104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46327028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15196122 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421940329414144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6492983258624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45630749 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541793437633065 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1537433167724544 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415446978064384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6492948729344 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967770 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15196361 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 $ _timestamp‚1.784328649722187e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1542466491826176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1542466491826176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46327028 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957777 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28923848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024712657449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46337015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚442 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1550384932106240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5077369600000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46754275 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29182997 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46744302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _runtime‚ 8980.83408635 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1555454958410439 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46754275 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1556132546926592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425795121376768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5078714989568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13852338 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1556132546926592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787124 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6551309336064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 $ _timestamp‚1.784328727459358e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419243812040704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46032849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚5399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46032843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551345969664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425795526101504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29183272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15330934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5068681703877 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceSVReâtatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46032843 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1555453588382149 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46032849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 B(›üO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787124 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6551309336064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 $ _timestamp‚1.784328727459358e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419243812040704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46032849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚5399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46032843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551345969664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425795526101504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29183272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15330934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5068681703877 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46032843 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1555453588382149 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46032849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚442 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1550384932106240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5077369600000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46754275 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29182997 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46744302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _runtime‚ 8980.83408635 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1555454958410439 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46754275 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1556132546926592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425795121376768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5078714989568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13852338 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1556132546926592ºÃCÝ2œ- _wandbruntime‚8982  _runtime‚8982³†öÛ 'ƒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5079640032256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1556411472692224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419321298448384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46041055 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46762767 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793829 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1551333178056704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1550662813884416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15333677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5079640032256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29188293 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803822 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552533018112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚8982.220422254 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46752780 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1551333178056704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46041049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13854881 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1555733763654973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843287288455439e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1556412844262400 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419321666539520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6552496391680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5069605170123 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1551333204230144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46752780 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1550662788456448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚443 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92803822 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1551333204230144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5070949770557 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5078294635520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640EY¨à'žˆN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46762767 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793829 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1551333178056704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1550662813884416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15333677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5079640032256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29188293 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803822 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552533018112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚8982.220422254 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46752780 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1551333178056704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46041049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13854881 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1555733763654973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843287288455439e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1556412844262400 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419321666539520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6552496391680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5069605170123 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1551333204230144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46752780 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1550662788456448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚443 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92803822 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1551333204230144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5070949770557 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5078294635520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5079640032256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1556411472692224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419321298448384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46041055 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5399»·4—2Ÿ- _wandbruntime‚8982  _runtime‚8982“ ÿ 0loss/FRt2/assignment_loss‚1.6996367531000542 1loss/FBt2/assignment_loss‚0.05416074504830993 .loss/FRt1/detection_loss‚1.391948428107775 1loss/SRqqt1/detection_loss‚0.6431437474066979 /loss/FBt1/detection_loss‚0.4928367716807721 _runtime‚8982.305433683 1loss/SRqqt2/detection_loss‚0.9033023432867837 %loss/total_loss‚8.520544085185367 epoch‚13 trainer/global_step‚5399 _step‚444 /loss/FRt1/assignment_loss‚0.713348622090506 1loss/FBt1/assignment_loss‚0.02888203560657526 $ _timestamp‚1.784328728931064e+09 2loss/SRqqt1/assignment_loss‚0.5502944423323601 2loss/SRqqt2/assignment_loss‚0.5410857709669561 /loss/FRt2/detection_loss‚0.5542107368325924 /loss/FBt2/detection_loss‚0.9476936887259837â9¬d¡„ 2loss/SRqqt2/assignment_loss‚0.5410857709669561 /loss/FRt2/detection_loss‚0.5542107368325924 /loss/FBt2/detection_loss‚0.9476936887259837 0loss/FRt2/assignment_loss‚1.6996367531000542 1loss/FBt2/assignment_loss‚0.05416074504830993 .loss/FRt1/detection_loss‚1.391948428107775 1loss/SRqqt1/detection_loss‚0.6431437474066979-ë¹Ì¦ /loss/FBt1/detection_loss‚0.4928367716807721 _runtime‚8982.305433683 1loss/SRqqt2/detection_loss‚0.9033023432867837 %loss/total_loss‚8.520544085185367 epoch‚13 trainer/global_step‚5399 _step‚444 /loss/FRt1/assignment_loss‚0.713348622090506 1loss/FBt1/assignment_loss‚0.02888203560657526 $ _timestamp‚1.784328728931064e+09 2loss/SRqqt1/assignment_loss‚0.5502944423323601¼‚”¢‚(:‰ ›äêÒÍÓ¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.9504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â>6a2£‚(:§ ›äêÒéš‘¯ disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda.out‚ 477.421875 disk.sda1.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚38.98771943122581 disk.sda.in‚ 5.93359375 disk.rbd2.out‚58.78564453125 disk.sda2.out‚0 disk.dm-1.out‚ 477.421875$ disk.nvme0c0n1.in‚19196.76953125" disk.nvme0n1.in‚19196.76953125! proc.memory.rssMB‚ 2950.078125 disk.dm-0.in‚19196.76953125 disk.sda1.out‚0 disk.sda3.in‚ 5.93359375# disk.nvme0n1.out‚72104.63671875 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.dm-0.out‚72104.63671875 disk.sda3.out‚ 477.421875) proc.memory.availableMB‚ 1945.87109375 proc.cpu.threads‚17 network.sent‚ 1.0177144e+07 disk.drbd2165.out‚0$ memory_percent‚88.12334537506104* proc.memory.percent‚18.005847930908203 network.recv‚ 2.305961e+06# disk.rbd1.out‚19354.43505859375 disk.dm-1.in‚ 5.93359375% disk.nvme0c0n1.out‚72104.63671875 cpu‚25.58556723807361$ disk./.usageGB‚580.9451942443848 disk.rbd0.in‚00œ³sž¤‚(:“ ªäêÒѮͬ gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.977' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.99539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹By <¥‚(:± ªäêÒ¬©‰® disk.sda2.in‚0 disk.rbd2.out‚58.78564453125% disk.nvme0n1.out‚72112.8818359375* proc.memory.percent‚18.005847930908203) disk./.usagePercent‚38.98771891921074" disk.dm-0.out‚72112.8818359375 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 477.66015625 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1945.85546875! proc.memory.rssMB‚ 2950.078125 proc.cpu.threads‚17 disk.sda.in‚ 5.93359375 disk.sda.out‚ 477.66015625# disk.rbd1.out‚19354.51318359375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.drbd2165.out‚0 disk.sda3.out‚ 477.66015625' disk.nvme0c0n1.out‚72112.8818359375 network.recv‚ 2.311136e+06 disk.dm-0.in‚ 19197.7265625 disk.sda1.out‚0 network.sent‚ 1.0239558e+07$ disk./.usageGB‚580.9451866149902 disk.rbd0.in‚0 disk.sda3.in‚ 5.93359375! disk.nvme0n1.in‚ 19197.7265625 cpu‚25.585265149266206# disk.nvme0c0n1.in‚ 19197.7265625 disk.sda1.in‚0$ memory_percent‚88.12344074249268׺vŒ”¦‚(:‰ ¹äêÒŠþ©¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(‰ Þ-§‚(:¢ ¹äêÒ˜œñ® disk.sda.out‚477.875 disk.rbd2.in‚0.015625# disk.dm-0.out‚72118.58447265625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375' proc.memory.availableMB‚ 1945.609375! proc.memory.rssMB‚ 2950.078125 network.sent‚ 1.0244613e+07 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19354.51318359375 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 cpu‚25.58492523578111 disk.rbd1.in‚ 506.52734375 disk.drbd2165.in‚0 disk.sda3.in‚ 5.93359375( disk.nvme0c0n1.out‚72118.58447265625$ memory_percent‚88.12494277954102 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚477.875) disk./.usagePercent‚38.98757811506691 disk.sda2.out‚0 disk.dm-0.in‚ 19197.7265625 disk.dm-1.out‚477.875# disk.nvme0c0n1.in‚ 19197.7265625 network.recv‚ 2.314659e+06* proc.memory.percent‚18.005847930908203$ disk./.usageGB‚580.9430885314941! disk.nvme0n1.in‚ 19197.7265625& disk.nvme0n1.out‚72118.58447265625 disk.rbd0.in‚0{žU”¨‚(:‰ ÈäêÒ¬Ùþ¬ gpu.0.gpu‚53.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚342.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.4228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/­ùô<©‚(:± ÈäêÒ”Ìç® disk.sda3.in‚ 5.93359375 disk.sda.in‚ 5.93359375& disk.nvme0n1.out‚72126.69873046875 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚25.58459173371807 disk.sda2.out‚0" disk.nvme0n1.in‚19263.73828125 disk.sda.out‚ 478.0859375( proc.memory.availableMB‚ 1945.6484375 disk.sda1.in‚0 network.sent‚ 1.025026e+07$ disk./.usageGB‚580.9431495666504$ disk.nvme0c0n1.in‚19263.73828125 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375! proc.memory.rssMB‚ 2950.078125 proc.cpu.threads‚17) disk./.usagePercent‚38.98758221118746$ memory_percent‚88.12470436096191 disk.rbd2.in‚0.015625 disk.dm-0.in‚19263.73828125# disk.dm-0.out‚72126.69873046875 disk.drbd2165.in‚0* proc.memory.percent‚18.005847930908203 disk.sda3.out‚ 478.0859375( disk.nvme0c0n1.out‚72126.69873046875# disk.rbd1.out‚19354.57568359375 disk.dm-1.out‚ 478.0859375 disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 network.recv‚ 2.317933e+06s‡ã”ª‚(:‰ ×äêÒÑÁ¶¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ„&;«‚(:° ×äêÒŒñˆ¯ disk.dm-1.out‚ 478.3984375$ disk.nvme0c0n1.in‚19263.73828125( disk.nvme0c0n1.out‚72132.01416015625! proc.memory.rssMB‚ 2950.078125 cpu‚25.584229799621834) disk./.usagePercent‚38.98761626018951 disk.sda.out‚ 478.3984375 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚72132.01416015625 disk.drbd2165.out‚0" disk.nvme0n1.in‚19263.73828125 network.recv‚ 2.321429e+06 disk.rbd2.out‚58.78564453125 disk.rbd2.in‚0.015625 disk.dm-0.in‚19263.73828125# disk.dm-0.out‚72132.01416015625% proc.memory.availableMB‚ 1946.0625$ disk./.usageGB‚580.9436569213867 disk.sda2.in‚0# disk.rbd1.out‚19354.57568359375 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 478.3984375$ memory_percent‚88.12217712402344 disk.sda.in‚ 5.93359375 network.sent‚ 1.0254782e+07 disk.sda1.in‚0 disk.sda3.in‚ 5.93359375* proc.memory.percent‚18.005847930908203 proc.cpu.threads‚17 disk.drbd2165.in‚0g•2¬- _wandbruntime‚9059  _runtime‚9059sIkŠÉ­Ã ' lr-AdamW/pg1‚0.00023158875396977867 ' lr-AdamW/pg2‚0.00023158875396977867 trainer/global_step‚5449 % _timestamp‚1.7843288060887437e+09 _runtime‚9059.463388486 _step‚445á?Î®È ' lr-AdamW/pg1‚0.00023158875396977867 ' lr-AdamW/pg2‚0.00023158875396977867 trainer/global_step‚5449 % _timestamp‚1.7843288060887437e+09 _runtime‚9059.463388486 _step‚445½a–ß2¯- _wandbruntime‚9059  _runtime‚9059¸(¯þ'°øO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46443149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1564955213832192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1570078899362816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1569395284432347 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564279021015040 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46443143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15468984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423118132424704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6610588049920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5123685530624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13979483 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564955213832192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5116263417307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5123685530624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1570078899362816 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29448472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784328806089894e+09 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564278995587072 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47170054 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚446 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47180027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47180027 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564955240005632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46443149 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5125030920192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0„\Ï(±ýO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5123685530624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1570078899362816 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29448472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784328806089894e+09 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564278995587072 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47170054 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚446 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47180027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47180027 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564955240005632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46443149 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5125030920192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46443149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1564955213832192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1570078899362816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1569395284432347 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564279021015040 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46443143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15468984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423118132424704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6610588049920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5123685530624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13979483 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564955213832192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5116263417307¾-Ñf¼•²‚(:Š æäêÒ˜ë׫ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a˜g7³‚(:¬ æäêÒ‹úµ¯! proc.memory.rssMB‚ 2950.078125 network.recv‚ 2.32481e+06 disk.rbd2.in‚0.015625 disk.dm-0.in‚ 19264.6953125 disk.sda2.out‚0( disk.nvme0c0n1.out‚72139.58251953125 disk.sda.out‚ 478.7265625 disk.sda1.out‚0 network.sent‚ 1.0259888e+07! disk.nvme0n1.in‚ 19264.6953125 disk.drbd2165.in‚0 disk.dm-1.out‚ 478.7265625 disk.sda3.in‚ 5.93359375* proc.memory.percent‚18.005847930908203# disk.dm-0.out‚72139.58251953125 disk.sda1.in‚0& disk.nvme0n1.out‚72139.58251953125 disk.rbd0.in‚0# memory_percent‚88.1220817565918' proc.memory.availableMB‚ 1946.078125# disk.rbd1.out‚19354.71630859375 disk.drbd2165.out‚0 disk.sda3.out‚ 478.7265625 cpu‚25.58392356052207# disk./.usageGB‚580.943603515625 disk.rbd2.out‚58.78564453125 proc.cpu.threads‚17 disk.dm-1.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375* disk./.usagePercent‚38.987612676084034# disk.nvme0c0n1.in‚ 19264.6953125 disk.sda.in‚ 5.93359375 disk.sda2.in‚0 disk.rbd0.out‚0R}2´- _wandbruntime‚9060  _runtime‚9060¸Ïƈ͵‚N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423195986923520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47188519 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29453760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5124610566144 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46451355 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565233240735744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429807389218304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5125955962880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13982031 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1570359196698624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629881 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15471729 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚5449 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565233214562304 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982026 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 $ _timestamp‚1.784328807475245e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629881 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46451355 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚447 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1565233214562304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5124610566144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1569674089676881 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1569672719648479 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1570357825128448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29453497 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6611770385920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6611807012352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429807793935872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1564556877365248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1565233240735744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5117186883665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47178532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1570359196698624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5115842283231 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46451349 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5125955962880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13982031 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1570359196698624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629881 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15471729 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚5449 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565233214562304 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982026 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 $ _timestamp‚1.784328807475245e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629881 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46451355 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚447 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1565233214562304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5124610566144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1569674089676881 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1569672719648479 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1570357825128448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29453497 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6611770385920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6611807012352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429807793935872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1564556877365248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1565233240735744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5117186883665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47178532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1570359196698624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5115842283231 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46451349 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EA¹€ 0loss/SRqqt1/detection_loss‚0.619835726975076 /loss/FBt2/detection_loss‚0.6021791594102979 $loss/total_loss‚8.22168814082801 epoch‚13 /loss/FRt2/assignment_loss‚1.652810378310895 2loss/SRqqt1/assignment_loss‚0.4757263771659521 .loss/FRt1/detection_loss‚1.400781971099064 1loss/SRqqt2/detection_loss‚0.9069389109021035 /loss/FBt1/detection_loss‚0.5528463788552733 2loss/SRqqt2/assignment_loss‚0.5287696693113981 /loss/FRt2/detection_loss‚0.6257001902972665 trainer/global_step‚5449 # _timestamp‚1.78432880756162e+09 _runtime‚9060.935953567 0loss/FRt1/assignment_loss‚0.8192954154458422 _step‚448 /loss/FBt1/assignment_loss‚0.033865616660869 2loss/FBt2/assignment_loss‚0.002938346393971728À”‚úT”º‚(:‰ õäêÒË´™­ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'_€9»‚(:® õäêÒ¶²Ë® disk.dm-1.out‚ 478.9609375 disk.sda3.out‚ 478.9609375 disk.sda.out‚ 478.9609375 disk.rbd1.in‚ 506.52734375 proc.cpu.threads‚17$ disk./.usageGB‚580.9452209472656 disk.sda1.out‚0 disk.sda2.out‚0) disk./.usagePercent‚38.98772122327855 disk.dm-0.in‚19265.52734375! disk.dm-0.out‚72148.435546875$ disk.nvme0n1.out‚72148.435546875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 5.93359375 network.recv‚ 2.329662e+06 disk.drbd2165.in‚0* proc.memory.percent‚18.005847930908203& disk.nvme0c0n1.out‚72148.435546875) proc.memory.availableMB‚ 1946.01953125 cpu‚25.583570387242393 network.sent‚ 1.0312811e+07 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375" disk.nvme0n1.in‚19265.52734375 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125! proc.memory.rssMB‚ 2950.078125$ disk.nvme0c0n1.in‚19265.52734375 disk.sda2.in‚0# disk.rbd1.out‚19354.71630859375$ memory_percent‚88.12243938446045 disk.sda1.in‚0í‘Vû ¼‚(:• „åêÒßçç« gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚298.104' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.620799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t£îA½‚(:¶ „åêÒ㺵® network.recv‚ 2.33407e+06 disk.rbd0.out‚0# disk.dm-0.out‚72152.38720703125 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2950.078125* proc.memory.percent‚18.005847930908203 disk.dm-0.in‚19364.77734375$ disk./.usageGB‚580.9457206726074 disk.sda.out‚ 479.23046875) proc.memory.availableMB‚ 1945.98828125 disk.sda1.out‚0 disk.sda3.out‚ 479.23046875( disk.nvme0c0n1.out‚72152.38720703125 disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda3.in‚ 5.93359375 disk.dm-1.out‚ 479.23046875 proc.cpu.threads‚17) disk./.usagePercent‚38.98775476026554 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚88.12263011932373& disk.nvme0n1.out‚72152.38720703125 disk.sda2.in‚0# disk.rbd1.out‚19354.77880859375 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.rbd2.out‚58.78564453125$ disk.nvme0c0n1.in‚19364.77734375" disk.nvme0n1.in‚19364.77734375 disk.sda.in‚ 5.93359375 cpu‚25.583269214881348 network.sent‚ 1.0364917e+074?+ž¾‚(:“ “åêÒöÁŽ® gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚299.641' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.92820000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êkŽI9¿‚(:® “åêÒԢȰ* disk./.usagePercent‚38.987751944182655 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625! proc.memory.rssMB‚ 2950.078125 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 479.5546875( disk.nvme0c0n1.out‚72163.57177734375# disk.rbd1.out‚19354.77880859375 disk.drbd2165.out‚0 disk.sda.in‚ 5.93359375 disk.rbd2.out‚58.78564453125* proc.memory.percent‚18.005847930908203 disk.sda3.in‚ 5.93359375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1945.98828125 network.sent‚ 1.0370521e+07 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 19365.734375 disk.nvme0n1.in‚ 19365.734375# disk.dm-0.out‚72163.57177734375$ memory_percent‚88.12263011932373 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚72163.57177734375 disk.sda2.in‚0 network.recv‚ 2.337362e+06$ disk./.usageGB‚580.9456787109375 disk.dm-1.out‚ 479.5546875 disk.dm-0.in‚ 19365.734375 cpu‚25.58295562826431 disk.sda1.in‚0 disk.sda.out‚ 479.5546875 disk.rbd0.out‚0ÆŠW”À‚(:‰ ¢åêÒÒǬ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²—‹Ö7Á‚(:¬ ¢åêÒÃÍõ®* proc.memory.percent‚18.005847930908203 disk.sda2.in‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚19366.56640625 disk.dm-1.out‚ 479.90625 cpu‚25.58263772117769# disk.dm-0.out‚72171.91650390625 disk.sda3.out‚ 479.90625" disk.nvme0n1.in‚19366.56640625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚19366.56640625 disk.sda.in‚ 5.93359375( proc.memory.availableMB‚ 1945.7421875# disk.rbd1.out‚19354.85693359375 disk.sda2.out‚0) disk./.usagePercent‚38.98777242478539 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375& disk.nvme0n1.out‚72171.91650390625$ memory_percent‚88.12413215637207 disk.rbd0.out‚0 disk.sda3.in‚ 5.93359375 network.sent‚ 1.0375207e+07 disk.rbd0.in‚0( disk.nvme0c0n1.out‚72171.91650390625! proc.memory.rssMB‚ 2950.078125 proc.cpu.threads‚17 network.recv‚ 2.340829e+06$ disk./.usageGB‚580.9459838867188 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda.out‚ 479.90625U®¶“‚(:ˆ ±åêÒ¡„ã« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.815' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.963 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[% %?Â(:´ ±åêÒ…ÀÇ®# disk.rbd1.out‚19354.85693359375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚19366.56640625' disk.nvme0c0n1.out‚72177.9580078125% disk.nvme0n1.out‚72177.9580078125! proc.memory.rssMB‚ 2950.078125* proc.memory.percent‚18.005847930908203$ disk./.usageGB‚580.9459533691406 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 480.08203125 disk.sda2.out‚0 network.sent‚ 1.0380279e+07 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.sda.out‚ 480.08203125 network.recv‚ 2.344561e+06) disk./.usagePercent‚38.98777037672512 disk.rbd0.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚19366.56640625 disk.sda.in‚ 5.93359375 disk.dm-0.in‚19366.56640625 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 480.08203125$ memory_percent‚88.12267780303955 cpu‚25.582286157346267 proc.cpu.threads‚17 disk.rbd2.out‚58.78564453125" disk.dm-0.out‚72177.9580078125 disk.drbd2165.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1945.98046875àhgu2Ä- _wandbruntime‚9138  _runtime‚9138äMîèÈÅ ' lr-AdamW/pg1‚0.00023079084364799045 ' lr-AdamW/pg2‚0.00023079084364799045 trainer/global_step‚5499 $ _timestamp‚1.784328884984042e+09 _runtime‚9138.358637573 _step‚449)sÍÆÇ ' lr-AdamW/pg1‚0.00023079084364799045 ' lr-AdamW/pg2‚0.00023079084364799045 trainer/global_step‚5499 $ _timestamp‚1.784328884984042e+09 _runtime‚9138.358637573 _step‚449Á”þ?2Ç- _wandbruntime‚9138  _runtime‚9138V86’ØÈùO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1584026623362048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426992452808704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15606640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1578855250337792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5171346850816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46853449 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1584025251799040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5170001461248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47605779 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433662860170752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1584026623362048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5171346850816 _step‚450 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1578855276511232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47595806 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5161155930093 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46853449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47595806 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5170001461248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1578855250337792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1583334240425965 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47605779 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29713273 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14106638 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426992820899840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46853443 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1584025251799040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433662457543168 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¤ªW¨(ÉþO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843288849851077e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1578173109923840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1578855276511232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6670039270912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1578855250337792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5171346850816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46853449 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1584025251799040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5170001461248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47605779 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433662860170752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1584026623362048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5171346850816 _step‚450 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1578855276511232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47595806 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5161155930093 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46853449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47595806 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5170001461248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1578855250337792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1583334240425965 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47605779 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29713273 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14106638 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426992820899840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46853443 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1584025251799040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433662457543168 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1584026623362048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426992452808704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15606640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47614271 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433741534085632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46861649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14109176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1584305549134848 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14109181 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 9139.75139107 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170926496768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579133251067904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170926496768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579133277241344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29718800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29718557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5163423996773 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6671226778112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46861655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1583613045670387 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1584305549134848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47604284 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427070307307520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1578450966274048 trainer/global_step‚5499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1578450991702016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5162079396339 _step‚451 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475926 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47604284 % _timestamp‚1.7843288863764734e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1583614415698789'ˆµ«ẋN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1578450991702016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5162079396339 _step‚451 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475926 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47604284 % _timestamp‚1.7843288863764734e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1583614415698789 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1584304177564672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46861649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46861655 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47614271 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433741534085632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46861649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14109176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1584305549134848 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14109181 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 9139.75139107 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170926496768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579133251067904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170926496768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579133277241344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚297188¢üÿ• 00 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29718557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5163423996773 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6671226778112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46861655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1583613045670387 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1584305549134848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47604284 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427070307307520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1578450966274048 trainer/global_step‚5499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280Ã=!¸I2Í- _wandbruntime‚9139  _runtime‚9139ÿ · /loss/FRt1/detection_loss‚1.4392281699387612 /loss/FRt2/detection_loss‚0.5706628427416623 1loss/SRqqt2/detection_loss‚0.9695567012211346 /loss/FBt1/detection_loss‚0.4964959634413632 0loss/FBt2/detection_loss‚0.44759547997576493 epoch‚13 trainer/global_step‚5499 1loss/SRqqt1/detection_loss‚0.6068390926158624 %loss/total_loss‚7.845101219452723 0loss/FRt2/assignment_loss‚1.7040725185006274 3loss/SRqqt1/assignment_loss‚0.48137936407789916 3loss/SRqqt2/assignment_loss‚0.48772989414497936 2loss/FBt2/assignment_loss‚0.022286790292176544 _runtime‚9139.831842506 0loss/FRt1/assignment_loss‚0.6078967801932669 % _timestamp‚1.7843288864573445e+09 _step‚452 2loss/FBt1/assignment_loss‚0.011357622309225945ÜwòNÏŒ 2loss/FBt1/assignment_loss‚0.011357622309225945 /loss/FRt1/detection_loss‚1.4392281699387612 /loss/FRt2/detection_loss‚0.5706628427416623 1loss/SRqqt2/detection_loss‚0.9695567012211346 /loss/FBt1/detection_loss‚0.4964959634413632 0loss/FBt2/detection_loss‚0.44759547997576493 epoch‚13 trainer/global_step‚5499 1loss/SRqqt1/detection_loss‚0.6068390926158624 %loss/total_loss‚7.845101219452723 0loss/FRt2/assignment_loss‚1.7040725185006274 3loss/SRqqt1/assignment_loss‚0.48137936407789916 3loss/SRqqt2/assignment_loss‚0.48772989414497936 2loss/FBt2/assignment_loss‚0.022286790292176544 _runtime‚9139.831842506 0loss/FRt1/assignment_loss‚0.6078967801932669 % _timestamp‚1.7843288864573445e+09 _step‚452ĺ”Ђ(:‰ ÀåêÒüóê« gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚343.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 qB5Ñ‚(:ª ÀåêÒÚÇÒ® disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 5.93359375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚19471.23046875% disk.nvme0c0n1.out‚72185.24609375& proc.memory.availableMB‚ 1945.59375 disk.rbd2.in‚0.015625# disk.nvme0n1.out‚72185.24609375 disk.sda.in‚ 5.93359375 disk.sda2.out‚0# disk.rbd1.out‚19354.91943359375$ memory_percent‚88.12503814697266! proc.memory.rssMB‚ 2950.078125$ disk./.usageGB‚580.9461402893066 disk.dm-1.out‚ 480.44140625 disk.sda3.out‚ 480.44140625$ disk.nvme0c0n1.in‚19471.23046875 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375* proc.memory.percent‚18.005847930908203 disk.drbd2165.in‚0 disk.rbd0.in‚0 cpu‚25.581987628696808 disk.sda.out‚ 480.44140625 disk.drbd2165.out‚0 network.sent‚ 1.0384997e+07" disk.nvme0n1.in‚19471.23046875 disk.dm-0.out‚72185.24609375 disk.dm-1.in‚ 5.93359375 network.recv‚ 2.347061e+06( disk./.usagePercent‚38.9877829210943 disk.rbd2.out‚58.78564453125!v¤Ò‚(:… ÏåêÒ¤úɬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Љ¨³@Ó‚(:µ ÏåêÒ烕®$ disk./.usageGB‚580.9462089538574 disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚19471.23046875& disk.nvme0n1.out‚72191.24169921875! proc.memory.rssMB‚ 2950.078125 disk.dm-1.in‚ 5.93359375 network.recv‚ 2.352617e+06 disk.sda.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19354.91943359375 disk.sda1.out‚0$ memory_percent‚88.12611103057861" disk.nvme0n1.in‚19471.23046875 disk.dm-1.out‚ 480.94921875 cpu‚25.58166617843309 disk.dm-0.in‚19471.23046875 disk.drbd2165.in‚0 disk.sda.out‚ 480.94921875 network.sent‚ 1.044814e+07) disk./.usagePercent‚38.98778752922992 disk.rbd2.out‚58.78564453125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625) proc.memory.availableMB‚ 1945.41796875 disk.sda1.in‚0* proc.memory.percent‚18.005847930908203 proc.cpu.threads‚17 disk.sda3.out‚ 480.94921875( disk.nvme0c0n1.out‚72191.24169921875 disk.sda2.out‚0# disk.dm-0.out‚72191.24169921875 disk.drbd2165.out‚0)°¡¦“Ô‚(:ˆ ÞåêÒ¡ý¥­ gpu.0.gpu‚59.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜›n2Õ‚(:§ ÞåêÒ°¬ï¯ disk.rbd0.in‚0 disk.dm-0.in‚ 19472.0625 disk.dm-1.in‚ 5.93359375 disk.nvme0c0n1.in‚ 19472.0625 disk.sda2.out‚0# disk.rbd1.out‚19354.99755859375 network.sent‚ 1.0452708e+07 disk.nvme0n1.in‚ 19472.0625% disk.nvme0n1.out‚72199.0166015625 disk.sda.out‚ 481.19140625$ memory_percent‚88.12623023986816$ disk./.usageGB‚580.9432983398438 disk.rbd2.out‚58.78564453125 disk.dm-1.out‚ 481.19140625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 481.19140625' disk.nvme0c0n1.out‚72199.0166015625 disk.sda.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375 cpu‚25.581305980151054 network.recv‚ 2.355889e+06 disk.sda1.in‚0( proc.memory.availableMB‚ 1945.3984375! proc.memory.rssMB‚ 2950.078125* proc.memory.percent‚18.005847930908203 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0" disk.dm-0.out‚72199.0166015625 disk.drbd2165.out‚0 disk.sda3.in‚ 5.93359375 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.98759219548129ÐÏn_”Ö‚(:‰ íåêÒù«¡¬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚244.269' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.8538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„†-™8ׂ(:­ íåêÒП¯& disk.nvme0c0n1.out‚72220.087890625 disk.sda.in‚ 5.93359375 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 482.23828125$ disk.nvme0n1.out‚72220.087890625 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚88.12522888183594! proc.memory.rssMB‚ 2950.078125 network.recv‚ 2.359584e+06) disk./.usagePercent‚38.98789479638676 disk.dm-0.in‚19473.01953125 disk.drbd2165.out‚0 network.sent‚ 1.0457971e+07! disk.dm-0.out‚72220.087890625" disk.nvme0n1.in‚19473.01953125 disk.sda2.in‚0% proc.memory.availableMB‚ 1945.5625* proc.memory.percent‚18.005847930908203$ disk.nvme0c0n1.in‚19473.01953125# disk.rbd1.out‚19354.99755859375$ disk./.usageGB‚580.9478073120117 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375 disk.sda.out‚ 482.23828125 disk.rbd1.in‚ 506.52734375 cpu‚25.581042417327456 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚ 482.23828125þÛ­U”Ø‚(:‰ üåêÒ€ú̬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚298.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.7264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ε½)Ù‚(:ž üåêÒü network.sent‚ 1.0463167e+07 network.recv‚ 2.362807e+06 disk.dm-0.in‚ 19607.546875 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 19607.546875 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375' disk.nvme0c0n1.out‚72224.6494140625% disk.nvme0n1.out‚72224.6494140625 cpu‚25.580694774031052$ disk./.usageGB‚580.9487762451172" disk.dm-0.out‚72224.6494140625 disk.sda.out‚483.5625 disk.sda2.in‚0% proc.memory.availableMB‚ 1945.5625! proc.memory.rssMB‚ 2950.078125 disk.rbd2.out‚58.78564453125 disk.dm-1.out‚483.5625 disk.sda.in‚ 5.93359375 proc.cpu.threads‚17) disk./.usagePercent‚38.98795982230046 disk.rbd2.in‚0.015625 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19355.12255859375* proc.memory.percent‚18.005847930908203 disk.rbd0.out‚0 disk.nvme0n1.in‚ 19607.546875 disk.sda2.out‚0$ memory_percent‚88.12522888183594 disk.rbd0.in‚0 disk.sda3.in‚ 5.93359375 disk.sda3.out‚483.5625 disk.sda1.in‚0GÁ42Ú- _wandbruntime‚9218  _runtime‚9218Í!w*ÉÛà ' lr-AdamW/pg1‚0.00022998691144826078 ' lr-AdamW/pg2‚0.00022998691144826078 trainer/global_step‚5549 % _timestamp‚1.7843289647544494e+09 _runtime‚9218.129083451 _step‚453zžYWÎÜÈ % _timestamp‚1.7843289647544494e+09 _runtime‚9218.129083451 _step‚453 ' lr-AdamW/pg1‚0.00022998691144826078 ' lr-AdamW/pg2‚0.00022998691144826078 trainer/global_step‚5549Åv†«.2Ý- _wandbruntime‚9218  _runtime‚92185Pe³þ'ÞøO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1592755313016832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15744182 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48031531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437596578999808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1597275936476163 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47263743 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1592067173404672 trainer/global_step‚5549 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1597971604235264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47263749 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14233788 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5217662781440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437596981627392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430866773192704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1597972975798272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48031531 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48021558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15743934 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729805807104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14233783 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5208737643523 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1592755313016832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47263749 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1592755286843392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29977717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1592755286843392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6729840343552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5216317391872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1597971604235264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29977970 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1592067198832640 _step‚454 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430867141283840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0W½(ßýO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48021558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15743934 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729805807104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14233783 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5208737643523 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1592755313016832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47263749 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1592755286843392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29977717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1592755286843392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6729840343552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5216317391872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1597971604235264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29977970 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1592067198832640 _step‚454 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430867141283840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1592755313016832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15744182 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48031531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437596578999808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1597275936476163 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47263743 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1592067173404672 trainer/global_step‚5549 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1597971604235264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47263749 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14233788 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5217662781440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437596981627392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430866773192704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1597972975798272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48031531Æâ602à- _wandbruntime‚9219  _runtime‚9219/^ wkáƒN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1593033313746944 OB€§¥îž&DeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29983244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430944259600384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47271949 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311978 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1592345055182848 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437675252741632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430944627691520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1593033313746944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48030036 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1593033287573504 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843289661476216e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5208316509447 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1597554741720697 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6731027670528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48040023 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437675655362048 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1597553371692295 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5218587824128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14236331 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15746913 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5218587824128 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5549 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47271955 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48040023 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5209661109881 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15746673 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1592345080610816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1593033287573504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6730993141248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14236326 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚9219.522493157 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311978 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5217242427392 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1598251901571072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29982999 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1598251901571072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95301985 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5549 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47271955 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48040023 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5209661109881 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15746673 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1592345080610816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1593033287573504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6730993141248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14236326 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚9219.522493157 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311978 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5217242427392 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1598251901571072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29982999 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1598251901571072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95301985 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311978 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1592345055182848 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437675252741632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430944627691520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1593033313746944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48030036 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1593033287573504 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843289661476216e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5208316509447 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1597554741720697 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820Ç–l¯z2ã- _wandbruntime‚9219  _runtime‚9219aÕ6Á ä† epoch‚13 trainer/global_step‚5549 % _timestamp‚1.7843289662288482e+09 2loss/SRqqt2/assignment_loss‚0.5497908644793349 /loss/FRt2/detection_loss‚0.5924598064485713 /loss/FBt1/detection_loss‚0.5116900960499897 /loss/FBt2/detection_loss‚0.7566684555883209 /loss/FRt1/detection_loss‚1.5284094902942138 1loss/SRqqt1/detection_loss‚0.6558401869515624 0loss/FRt1/assignment_loss‚1.3993166149486596 2loss/FBt1/assignment_loss‚0.040698870886195095 _runtime‚9219.603393301 _step‚456 0loss/FRt2/assignment_loss‚1.7853518937355386 2loss/SRqqt1/assignment_loss‚0.5421886169683797 4loss/FBt2/assignment_loss‚0.00043240552482421373 1loss/SRqqt2/detection_loss‚0.8807758427889436 %loss/total_loss‚9.243623144664534ž8[å‹ 1loss/SRqqt2/detection_loss‚0.8807758427889436 %loss/total_loss‚9.243623144664534 epoch‚13 trainer/global_step‚5549 % _timestamp‚1.7843289662288482e+09 2loss/SRqqt2/assignment_loss‚0.5497908644793349 /loss/FRt2/detection_loss‚0.5924598064485713 /loss/FBt1/detection_loss‚0.5116900960499897 /loss/FBt2/detection_loss‚0.7566684555883209 /loss/FRt1/detection_loss‚1.5284094902942138 1loss/SRqqt1/detection_loss‚0.6558401869515624 0loss/FRt1/assignment_loss‚1.3993166149486596 2loss/FBt1/assignment_loss‚0.040698870886195095 _runtime‚9219.603393301 _step‚456 0loss/FRt2/assignment_loss‚1.7853518937355386 2loss/SRqqt1/assignment_loss‚0.5421886169683797 4loss/FBt2/assignment_loss‚0.00043240552482421373ÈKñ žæ‚(:“ ‹æêÒ¸ô°¬ gpu.0.gpu‚47.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚321.537' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.30739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ù9ç‚(:® ‹æêÒÛ̹¯* proc.memory.percent‚18.005847930908203 network.sent‚ 1.0468232e+07$ disk.nvme0n1.out‚72237.689453125 proc.cpu.threads‚17 network.recv‚ 2.366135e+06) disk./.usagePercent‚38.98822837420388$ disk./.usageGB‚580.9527778625488 disk.rbd2.out‚58.78564453125! disk.dm-0.out‚72237.689453125! proc.memory.rssMB‚ 2950.078125 disk.drbd2165.out‚0 disk.dm-1.out‚ 484.21484375 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.sda1.out‚0" disk.nvme0n1.in‚19608.37890625 disk.sda.in‚ 5.93359375 disk.sda.out‚ 484.21484375 disk.rbd0.in‚0 disk.dm-0.in‚19608.37890625 disk.dm-1.in‚ 5.93359375$ disk.nvme0c0n1.in‚19608.37890625& disk.nvme0c0n1.out‚72237.689453125 disk.sda2.out‚0# disk.rbd1.out‚19355.12255859375 disk.sda1.in‚0 disk.sda3.in‚ 5.93359375 cpu‚25.580426317115826 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 484.21484375# memory_percent‚88.1281852722168' proc.memory.availableMB‚ 1945.078125jù0”è‚(:‰ šæêÒ™”ˬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ávžu>é‚(:³ šæêÒÍøø®) disk./.usagePercent‚38.98794881397649 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1945.07421875 network.recv‚ 2.371342e+06 disk.sda3.in‚ 5.93359375 disk.sda2.in‚0 network.sent‚ 1.0531395e+07( disk.nvme0c0n1.out‚72245.88916015625# disk.dm-0.out‚72245.88916015625 disk.dm-1.out‚ 485.21484375* proc.memory.percent‚18.005847930908203 disk.sda.out‚ 485.21484375 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375$ disk./.usageGB‚580.9486122131348 disk.rbd1.in‚ 506.52734375! proc.memory.rssMB‚ 2950.078125 cpu‚25.580083119642353 proc.cpu.threads‚17 disk.sda3.out‚ 485.21484375# disk.nvme0c0n1.in‚ 19609.3359375& disk.nvme0n1.out‚72245.88916015625# memory_percent‚88.1282091140747# disk.rbd1.out‚19355.20068359375 disk.rbd0.out‚0 disk.dm-0.in‚ 19609.3359375 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 19609.3359375 disk.sda.in‚ 5.93359375 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0§püŸê‚(:” ©æêÒ™ï÷¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚140.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.044200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0piR=ë‚(:² ©æêÒªâ¯# disk.nvme0c0n1.in‚ 19609.3359375! disk.nvme0n1.in‚ 19609.3359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚72257.90771484375 network.sent‚ 1.0536466e+07 disk.sda.out‚ 486.59765625 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375$ disk./.usageGB‚580.9485626220703 disk.rbd0.out‚0$ memory_percent‚88.12699317932129( proc.memory.availableMB‚ 1945.2734375! proc.memory.rssMB‚ 2950.078125 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 19609.3359375 disk.sda3.out‚ 486.59765625& disk.nvme0n1.out‚72257.90771484375 disk.rbd2.in‚0.015625# disk.dm-0.out‚72257.90771484375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375* proc.memory.percent‚18.005847930908203 cpu‚25.57981426372392 network.recv‚ 2.375004e+06) disk./.usagePercent‚38.98794548587855# disk.rbd1.out‚19355.20068359375 disk.sda3.in‚ 5.93359375 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 486.59765625 proc.cpu.threads‚17¢ù©ˆžì‚(:“ ¸æêÒÏ‚ú« gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f©<ö>í‚(:³ ¸æêÒŽ€Þ­ disk.sda1.out‚0& disk.nvme0n1.out‚72265.70654296875 disk.sda.out‚ 487.18359375 disk.rbd2.in‚0.015625# disk.dm-0.out‚72265.70654296875! proc.memory.rssMB‚ 2950.078125) disk./.usagePercent‚38.98794394983334 disk.dm-1.in‚ 5.93359375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚ 506.52734375 network.sent‚ 1.0541574e+07 disk.dm-1.out‚ 487.18359375 disk.sda.in‚ 5.93359375 disk.sda1.in‚0 disk.sda3.in‚ 5.93359375( disk.nvme0c0n1.out‚72265.70654296875! disk.nvme0n1.in‚ 19782.1640625 disk.sda2.out‚0# disk.rbd1.out‚19355.26318359375$ memory_percent‚88.12544345855713 cpu‚25.57948195869524 network.recv‚ 2.378433e+06 disk.drbd2165.in‚0 disk.sda3.out‚ 487.18359375* proc.memory.percent‚18.005847930908203$ disk./.usageGB‚580.9485397338867 disk.drbd2165.out‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 19782.1640625 disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 19782.1640625) proc.memory.availableMB‚ 1945.52734375Àa‚”î‚(:‰ ÇæêÒÎàæ­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&D¤@ï‚(:µ ÇæêÒË÷Á°# disk.dm-0.out‚72277.62158203125 disk.sda1.out‚0 disk.drbd2165.out‚0 network.sent‚ 1.0547237e+07$ disk.nvme0c0n1.in‚19783.12109375( disk.nvme0c0n1.out‚72277.62158203125 disk.sda.out‚ 487.47265625 disk.sda2.out‚0$ memory_percent‚88.11793327331543( proc.memory.availableMB‚ 1946.7578125 network.recv‚ 2.382071e+06 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 487.47265625" disk.nvme0n1.in‚19783.12109375 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0$ disk./.usageGB‚580.9472236633301# disk.rbd1.out‚19355.26318359375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2950.078125 cpu‚25.5791768857191 disk.dm-1.out‚ 487.47265625 disk.rbd2.out‚58.78564453125& disk.nvme0n1.out‚72277.62158203125* proc.memory.percent‚18.005847930908203 disk.sda3.in‚ 5.93359375 disk.sda.in‚ 5.93359375 proc.cpu.threads‚17* disk./.usagePercent‚38.987855627234026 disk.rbd2.in‚0.015625 disk.dm-0.in‚19783.12109375"â»2ð- _wandbruntime‚9297  _runtime‚9297[‡éÇñÁ _step‚457 & lr-AdamW/pg1‚0.0002291770090056526 & lr-AdamW/pg2‚0.0002291770090056526 trainer/global_step‚5599 % _timestamp‚1.7843290445427625e+09 _runtime‚9297.918408014SÚÓ­ÌòÆ & lr-AdamW/pg2‚0.0002291770090056526 trainer/global_step‚5599 % _timestamp‚1.7843290445427625e+09 _runtime‚9297.918408014 _step‚457 & lr-AdamW/pg1‚0.0002291770090056526ÉõU{ñ2ó- _wandbruntime‚9297  _runtime‚9297ùÒБM ôøO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1611216262498071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1605961287741440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14360938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434741093576704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47674049 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5263978712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47674049 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚9297.918519365 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48447310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 IDeviceStatsMonitor.on_train_batKVü^±ch_start/active_bytes.all.freed‚1611917956671488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1606655323348992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚5599 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5262633322496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1611919328234496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1606655349522432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5262633322496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1606655323348992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5254974756631 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47674043 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47674043 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48457283 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14360933 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15880642 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441530763680256 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5253630156309 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1611214892469781 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1606655349522432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚458 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48457283 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30241575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1611917956671488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1606655323348992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚5599 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5262633322496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1611919328234496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1606655349522432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5262633322496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1606655323348992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5254974756631 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47674043 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47674043 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48457283 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14360933 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15880642 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441530763680256 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5253630156309 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1611214892469781 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1606655349522432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚458 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48457283 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30241575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1605961287741440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14360938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434741093576704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ê ªÐ2ö- _wandbruntime‚9299  _runtime‚9299¯ “I '÷ƒN C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138037 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48455788 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1611495067742605 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1611493697714203 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30246876 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434818579984384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1606239144091648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434818948075520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1612196882437120 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5254553622555 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1612198254007296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47682249 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _runtime‚9299.311687761 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5263558358016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15883400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441609862300160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5255898222989 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47682255 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5599 _step‚459 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14363481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14363476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1606933350252544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30247117 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6790881792512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1612198254007296 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 % _timestamp‚1.7843290459367526e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148030 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47682249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1606933324079104 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48465775 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 % _timestamp‚1.7843290459367526e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148030 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47682249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1606933324079104 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48465775 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1611495067742605 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1611493697714203 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚….m 30246876 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434818579984384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1606239144091648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434818948075520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1612196882437120 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5254553622555 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1612198254007296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47682249 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _runtime‚9299.311687761 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5263558358016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15883400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441609862300160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5255898222989 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47682255 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5599 _step‚459 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14363481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14363476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1606933350252544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30247117 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6790881792512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995Ëìk2ù- _wandbruntime‚9299  _runtime‚9299ûÁú 0loss/FRt2/assignment_loss‚1.8007778780327903 2loss/SRqqt2/assignment_loss‚0.5594571481664516 /loss/FBt2/detection_loss‚0.7474266548233572 $loss/total_loss‚8.98464291398388 trainer/global_step‚5599 2loss/FBt1/assignment_loss‚0.039132926729233374 1loss/FBt2/assignment_loss‚0.03623510001109698 /loss/FRt2/detection_loss‚0.5299131587149734 epoch‚13 _runtime‚9299.394039155 _step‚460 /loss/FRt1/detection_loss‚1.6912271112203598 1loss/SRqqt2/detection_loss‚0.8589343214644842 $ _timestamp‚1.784329046019443e+09 2loss/SRqqt1/assignment_loss‚0.5413020556220346 1loss/SRqqt1/detection_loss‚0.6248709555698222 /loss/FBt1/detection_loss‚0.4050724602454076 0loss/FRt1/assignment_loss‚1.1502931433838683ýjÒ û† /loss/FBt1/detection_loss‚0.4050724602454076 0loss/FRt1/assignment_loss‚1.1502931433838683 0loss/FRt2/assignment_loss‚1.8007778780327903 2loss/SRqqt2/assignment_loss‚0.5594571481664516 /loss/FBt2/detection_loss‚0.7474266548233572 $loss/total_loss‚8.98464291398388 trainer/global_step‚5599 2loss/FBt1/assignment_loss‚0.039132926729233374 1loss/FBt2/assignment_loss‚0.03623510001109698 /loss/FRt2/detection_loss‚0.5299131587149734 epoch‚13 _runtime‚9299.394039155 _step‚460 /loss/FRt1/detection_loss‚1.6912271112203598 1loss/SRqqt2/detection_loss‚0.8589343214644842 $ _timestamp‚1.784329046019443e+09 2loss/SRqqt1/assignment_loss‚0.5413020556220346 1loss/SRqqt1/detection_loss‚0.6248709555698222ÌØâm”ü‚(:‰ ÖæêÒµâñ« gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì"#@ý‚(:µ ÖæêÒà蘰 disk.dm-1.out‚ 488.48046875 disk.sda3.out‚ 488.48046875 disk.sda.out‚ 488.48046875) disk./.usagePercent‚38.98785588324156 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚72281.90966796875 cpu‚25.578890506280317 network.sent‚ 1.0551974e+07 disk.rbd2.out‚58.78564453125$ disk.nvme0c0n1.in‚19783.12109375" disk.nvme0n1.in‚19783.12109375 disk.rbd1.in‚ 506.52734375 network.recv‚ 2.385595e+06 disk.rbd2.in‚0.015625 disk.dm-0.in‚19783.12109375 disk.sda1.out‚0& disk.nvme0n1.out‚72281.90966796875 disk.sda2.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚72281.90966796875 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375* proc.memory.percent‚18.005847930908203$ disk./.usageGB‚580.9472274780273 disk.drbd2165.in‚0 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19355.34130859375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚88.11960220336914' proc.memory.availableMB‚ 1946.484375! proc.memory.rssMB‚ 2950.078125 proc.cpu.threads‚17æn]þ‚(:’ åæêÒÝ£÷« gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùi4]4ÿ‚(:© åæêÒû—ò®" disk.nvme0c0n1.in‚ 19783.953125$ disk.nvme0n1.out‚72292.587890625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 disk.dm-1.in‚ 5.93359375 disk.nvme0n1.in‚ 19783.953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 19783.953125* proc.memory.percent‚18.013906478881836 network.recv‚ 2.391179e+06$ memory_percent‚88.12873363494873) proc.memory.availableMB‚ 1944.98828125" proc.memory.rssMB‚ 2951.3984375 network.sent‚ 1.0615679e+07 disk.sda3.in‚ 5.93359375 disk.sda.in‚ 5.93359375 disk.sda3.out‚ 488.8046875 disk.sda.out‚ 488.8046875# disk.rbd1.out‚19355.34130859375 cpu‚25.583187727420075 proc.cpu.threads‚17$ disk./.usageGB‚580.9434204101562& disk.nvme0c0n1.out‚72292.587890625 disk.rbd0.out‚0 disk.rbd2.in‚0.015625! disk.dm-0.out‚72292.587890625 disk.drbd2165.out‚0 disk.dm-1.out‚ 488.8046875) disk./.usagePercent‚38.98760038772239 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375 disk.drbd2165.in‚0Õ€B2€- _wandbruntime‚9326  _runtime‚9326‡>…ξ=¸{ =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6219766139984131 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3672264814376831 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8560845255851746 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6967454552650452 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6232181191444397 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7154096961021423 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14090754091739655 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1407867968082428 .particle/accuracy_3_of_4‚0.997619092464447 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905720949172974 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9233363270759583 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975273609161377 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15241090953350067 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6629775166511536 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6225172281265259 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12445534020662308 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027651723474264145 +particle/sensitivity‚0.6176333427429199 /particle/accuracy_4_of_4‚0.9333785176277161 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969676733016968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6251077651977539 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13830965757369995 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1592978686094284 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6655413508415222 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9956160187721252 *jet/accuracy_2_of_4‚0.8012514710426331 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07362190634012222 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +particle/specificity‚0.9120590686798096 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4261137545108795 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16655108332633972 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 trainer/global_step‚5599 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7666416168212891 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6784958839416504 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061835236847400665 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915906190872192 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7583144903182983 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16316020488739014 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027739981189370155 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7241953015327454 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5060362815856934 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16236050426959991 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07852090150117874 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824074625968933 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.821892261505127 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16205373406410217 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚8.994350874580965 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4975000023841858 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 particle/accuracy_3_of_3‚1 % _timestamp‚1.7843290732487621e+09 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7485260963439941 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8733242750167847 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6779836416244507 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014780187979340553 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9928439855575562 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4193841814994812 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7027387619018555 jet/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7222175598144531 +jet/accuracy_1_of_1‚0.29991140961647034 'particle/f_score‚0.6369768977165222 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28278255462646484 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6764131188392639 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.174311563372612 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7146053314208984 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887006878852844 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15071742236614227 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4508472979068756 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6204816102981567 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6713988184928894 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6485050320625305 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5692577958106995 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5823681950569153 +jet/accuracy_3_of_4‚0.08534525334835052 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21057766675949097 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940986633300781 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15425840020179749 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3648989796638489 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7112506031990051 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.156809002161026 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6482260227203369 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5972874164581299 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 epoch‚13 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2313205599784851 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07965302467346191 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6657747030258179 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17533989250659943 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03730173781514168 jet/accuracy_2_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061835236847400665 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982857704162598 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6validation_average_jet_accuracy‚0.3477703630924225 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6190641522407532 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968254566192627 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1546049565076828 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7489607334136963 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 _runtime‚9326.624009912 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.596400260925293 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856827855110168 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07420653104782104 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7275239825248718 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6232181191444397 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1750316321849823 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7430083155632019 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014973646029829979 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3565763831138611 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_3‚0.9220686554908752 *jet/accuracy_2_of_3‚0.5224453806877136 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16316020488739014 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5786528587341309 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3685134947299957 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43594440817832947 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9856961369514465 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7685664892196655 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7008492946624756 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16875389218330383 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.821892261505127 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958730697632 _step‚461 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13498814404010773 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15401820838451385 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7821575403213501 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975273609161377 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921150207519531 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7110220789909363 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16668608784675598 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00821627862751484 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22407479584217072 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9167822003364563 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.664142370223999 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954330921173096 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.985876202583313 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6132512092590332 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5108559727668762 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.755047619342804 *jet/accuracy_1_of_4‚0.9634929895401001 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8488304615020752 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.673251748085022 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.750453531742096 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16413651406764984 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24372383952140808 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962281584739685 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16139501333236694 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9811443090438843 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1979166716337204 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24753084778785706 +jet/accuracy_3_of_3‚0.06306483596563339 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8834847211837769 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7367518544197083 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944920539855957 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1Á£}6¹‚½{ =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7008492946624756 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16875389218330383 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.821892261505127 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958730697632 _step‚461 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13498814404010773 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15401820838451385 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7821575403213501 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975273609161377 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921150207519531 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7110220789909363 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16668608784675598 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00821627862751484 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22407479584217072 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9167822003364563 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.664142370223999 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954330921173096 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.985876202583313 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6132512092590332 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5108559727668762 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.755047619342804 *jet/accuracy_1_of_4‚0.9634929895401001 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8488304615020752 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.673251748085022 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.750453531742096 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16413651406764984 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24372383952140808 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962281584739685 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16139501333236694 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9811443090438843 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1979166716337204 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24753084778785706 +jet/accuracy_3_of_3‚0.06306483596563339 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8834847211837769 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7367518544197083 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944920539855957 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6219766139984131 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3672264814376831 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8560845255851746 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6967454552650452 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6232181191444397 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7154096961021423 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14090754091739655 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1407867968082428 .particle/accuracy_3_of_4‚0.997619092464447 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905720949172974 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9233363270759583 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975273609161377 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15241090953350067 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6629775166511536 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6225172281265259 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12445534020662308 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027651723474264145 +particle/sensitivity‚0.6176333427429199 /particle/accuracy_4_of_4‚0.9333785176277161 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969676733016968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6251077651977539 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13830965757369995 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1592978686094284 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6655413508415222 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9956160187721252 *jet/accuracy_2_of_4‚0.8012514710426331 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07362190634012222 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +particle/specificity‚0.9120590686798096 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4261137545108795 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16655108332633972 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 trainer/global_step‚5599 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7666416168212891 ,&P>‚ur urity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6784958839416504 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061835236847400665 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915906190872192 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7583144903182983 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16316020488739014 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027739981189370155 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7241953015327454 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5060362815856934 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16236050426959991 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07852090150117874 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824074625968933 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.821892261505127 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16205373406410217 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚8.994350874580965 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4975000023841858 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 particle/accuracy_3_of_3‚1 % _timestamp‚1.7843290732487621e+09 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7485260963439941 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8733242750167847 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6779836416244507 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014780187979340553 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9928439855575562 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4193841814994812 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7027387619018555 jet/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7222175598144531 +jet/accuracy_1_of_1‚0.29991140961647034 'particle/f_score‚0.6369768977165222 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28278255462646484 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6764131188392639 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.174311563372612 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7146053314208984 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887006878852844 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15071742236614227 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4508472979068756 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6204816102981567 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6713988184928894 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6485050320625305 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5692577958106995 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5823681950569153 +jet/accuracy_3_of_4‚0.08534525334835052 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21057766675949097 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940986633300781 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15425840020179749 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3648989796638489 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7112506031990051 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.156809002161026 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6482260227203369 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5972874164581299 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 epoch‚13 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2313205599784851 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07965302467346191 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6657747030258179 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17533989250659943 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03730173781514168 jet/accuracy_2_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061835236847400665 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982857704162598 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6validation_average_jet_accuracy‚0.3477703630924225 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6190641522407532 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968254566192627 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1546049565076828 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7489607334136963 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 _runtime‚9326.624009912 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.596400260925293 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856827855110168 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07420653104782104 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7275239825248718 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6232181191444397 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1750316321849823 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7430083155632019 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014973646029829979 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3565763831138611 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_3‚0.9220686554908752 *jet/accuracy_2_of_3‚0.5224453806877136 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16316020488739014 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5786528587341309 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3685134947299957 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43594440817832947 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9856961369514465 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7685664892196655 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1Í–’ƒ‚(:‡ ôæêÒŽ³¬ gpu.0.gpu‚28.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚30.0 gpu.0.powerWatts‚91.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3592 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëüžA„‚(:¶ ôæêÒ´ý‘® disk.sda2.out‚0 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 488.98828125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚ 5.93359375 disk.sda2.in‚0 disk.dm-1.in‚ 5.93359375 cpu‚25.58623883548106 disk.rbd0.out‚0 disk.dm-0.in‚19796.34765625 disk.sda.out‚ 488.98828125 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚19796.34765625$ memory_percent‚84.74912643432617' proc.memory.availableMB‚ 2498.703125* proc.memory.percent‚20.599055290222168$ disk./.usageGB‚580.9511604309082# disk.rbd1.out‚19355.40380859375 disk.rbd2.out‚58.78564453125 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 488.98828125$ disk.nvme0c0n1.in‚19796.34765625 network.sent‚ 1.062076e+07* disk./.usagePercent‚38.988119827009356# disk.dm-0.out‚72296.14208984375 disk.drbd2165.out‚0& disk.nvme0n1.out‚72296.14208984375# proc.memory.rssMB‚ 3374.94921875 network.recv‚ 2.394536e+06( disk.nvme0c0n1.out‚72296.14208984375 proc.cpu.threads‚17¢'â>”…‚(:‰ ƒçêÒŽµŸ¬ gpu.0.gpu‚75.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+å¶9†‚(:® ƒçêÒýïõ® disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 489.21875 network.sent‚ 1.0677953e+07 disk.dm-0.in‚19877.27734375 disk.sda1.in‚0* proc.memory.percent‚18.045878410339355" disk.dm-0.out‚73275.0068359375 network.recv‚ 2.399469e+06 disk.rbd2.out‚58.78564453125 disk.sda.out‚ 489.21875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2916.55859375 disk.sda3.out‚ 489.21875 disk.sda2.out‚0# disk.rbd1.out‚19355.40380859375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚82.19873905181885$ disk.nvme0c0n1.in‚19877.27734375 disk.rbd2.in‚0.015625 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚580.9512214660645' disk.nvme0c0n1.out‚73275.0068359375" disk.nvme0n1.in‚19877.27734375% disk.nvme0n1.out‚73275.0068359375 disk.sda.in‚ 5.93359375 disk.sda3.in‚ 5.93359375* disk./.usagePercent‚38.988123923129905 disk.rbd1.in‚ 506.52734375# proc.memory.rssMB‚ 2956.63671875 cpu‚25.585055729201706,½™”‡‚(:‰ ’çêÒ¶Öº¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n˜ç33ˆ‚(:¨ ’çêÒ‘ ®# proc.memory.rssMB‚ 2956.63671875 cpu‚25.58527004747809 proc.cpu.threads‚17 network.recv‚ 2.402686e+06# disk.dm-0.out‚74252.03662109375 disk.sda1.out‚0 network.sent‚ 1.0683066e+07 disk.sda1.in‚0 disk.nvme0n1.in‚ 19878.109375* proc.memory.percent‚18.045878410339355& disk.nvme0n1.out‚74252.03662109375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 489.65625 disk.sda3.out‚ 489.65625$ memory_percent‚82.19876289367676) disk./.usagePercent‚38.98788122798745 disk.rbd1.in‚ 506.52734375" disk.nvme0c0n1.in‚ 19878.109375( proc.memory.availableMB‚ 2916.5546875$ disk./.usageGB‚580.9476051330566 disk.sda.in‚ 5.93359375 disk.rbd0.in‚0 disk.drbd2165.out‚0# disk.rbd1.out‚19355.54443359375 disk.sda.out‚ 489.65625 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375( disk.nvme0c0n1.out‚74252.03662109375 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 19878.109375Y2Ñc”‰‚(:‰ ¡çêÒÁöÙ¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÏ:Š‚(:¯ ¡çêÒÎþ® network.recv‚ 2.406098e+06& disk.nvme0n1.out‚74256.74853515625 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0 cpu‚25.58544885216243) disk./.usagePercent‚38.98786074738471$ disk./.usageGB‚580.9472999572754 disk.sda3.in‚ 5.93359375 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0) proc.memory.availableMB‚ 2916.80078125* proc.memory.percent‚18.045878410339355" disk.nvme0c0n1.in‚ 19878.109375 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0# disk.dm-0.out‚74256.74853515625# disk.rbd1.out‚19355.54443359375 disk.dm-0.in‚ 19878.109375 disk.dm-1.in‚ 5.93359375$ memory_percent‚82.19726085662842 disk.sda.in‚ 5.93359375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2956.63671875 disk.nvme0n1.in‚ 19878.109375 disk.sda.out‚ 489.8515625 disk.sda2.in‚0 disk.dm-1.out‚ 489.8515625 proc.cpu.threads‚17 network.sent‚ 1.0688127e+07 disk.sda3.out‚ 489.8515625( disk.nvme0c0n1.out‚74256.74853515625 disk.sda2.out‚0 disk.drbd2165.out‚01"‰’‹‚(:‡ °çêÒ—Œæ¬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P½DŒ‚(:¹ °çêÒÿœ¯ proc.cpu.threads‚17 network.sent‚ 1.0693227e+07 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 489.99609375( proc.memory.availableMB‚ 2916.3203125* disk./.usagePercent‚38.987859979362106 disk.sda.out‚ 489.99609375 disk.sda2.in‚0# disk.rbd1.out‚19355.60693359375 disk.rbd0.out‚0# disk.dm-0.out‚74265.12353515625* proc.memory.percent‚18.045878410339355 disk.rbd0.in‚0 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 489.99609375 disk.dm-0.in‚19879.06640625( disk.nvme0c0n1.out‚74265.12353515625 cpu‚25.585709638765337 network.recv‚ 2.409562e+06$ disk.nvme0c0n1.in‚19879.06640625" disk.nvme0n1.in‚19879.06640625$ disk./.usageGB‚580.9472885131836& disk.nvme0n1.out‚74265.12353515625 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.dm-1.in‚ 5.93359375$ memory_percent‚82.20019340515137# proc.memory.rssMB‚ 2956.63671875 ±¦”‚(:‰ ¿çêÒÿÙË­ gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Àsc3Ž‚(:¨ ¿çêÒÞÙ®$ disk./.usageGB‚580.9474716186523 disk.sda.out‚ 490.359375 disk.rbd2.in‚0.015625 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 490.359375( proc.memory.availableMB‚ 2916.3828125* proc.memory.percent‚18.045878410339355 network.sent‚ 1.0698391e+07 disk.dm-0.in‚ 20004.203125 disk.sda1.out‚0 disk.drbd2165.in‚0# memory_percent‚82.1998119354248 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 490.359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.585867890656004) disk./.usagePercent‚38.98787226772375 disk.rbd0.out‚0% disk.nvme0n1.out‚74275.1630859375 disk.sda.in‚ 5.93359375' disk.nvme0c0n1.out‚74275.1630859375 disk.rbd2.out‚58.78564453125 disk.drbd2165.out‚0 network.recv‚ 2.413329e+06" disk.nvme0c0n1.in‚ 20004.203125# proc.memory.rssMB‚ 2956.63671875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 20004.203125 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19355.60693359375" disk.dm-0.out‚74275.16308593755ít¤2- _wandbruntime‚9410  _runtime‚9410™ouÇÁ trainer/global_step‚5649 % _timestamp‚1.7843291570768838e+09 _runtime‚9410.451555801 _step‚462 & lr-AdamW/pg1‚0.0002283611883386863 & lr-AdamW/pg2‚0.0002283611883386863R¹RÌ‘Æ _runtime‚9410.451555801 _step‚462 & lr-AdamW/pg1‚0.0002283611883386863 & lr-AdamW/pg2‚0.0002283611883386863 trainer/global_step‚5649 % _timestamp‚1.7843291570768838e+09ÎÊŇB2’- _wandbruntime‚9410  _runtime‚9410,pHÿ'“ùO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629698133156864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16053789 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5316137256960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438663526207488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48216349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48942022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16054039 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14497001 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14497006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158377 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6857279300608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1624383367462912 _step‚463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628988326584029 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1624383341289472 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1624383367462912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6857313837056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9410.452755756 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30550790 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445521208135680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438663894298624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629699504719872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48942022 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148398 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5314791867392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445520805508096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5305696400859 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1624383341289472 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1623681260154880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5307041001181 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚5649 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1628986956555739 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30551045 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843291570779846e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438663526207488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48216349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48942022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16054039 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14497001 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14497006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158377 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6857279300608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1624383367462912 _step‚463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628988326584029 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1624383341289472 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1624383367462912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6857313837056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9410.452755756 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30550790 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445521208135680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438663894298624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629699504719872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48942022 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148398 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5314791867392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445520805÷îì[508096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5305696400859 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1624383341289472 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1623681260154880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5307041001181 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚5649 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1628986956555739 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30551045 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843291570779846e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629698133156864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16053789 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128ÏáJ2•- _wandbruntime‚9411  _runtime‚9411u«•L'–‚N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629978430492672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14499549 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1623959141933056 trainer/global_step‚5649 _step‚464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5317062299648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629978430492672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629977058922496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48950514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5306619867105 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48224561 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438741012615168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5307964467539 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5315716902912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445599494962176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚9411.844851916 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438741380706304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6858482347008 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48224561 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1624661368193024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30556091 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1629267131828563 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1624661368193024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175075 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48940527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14499544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48950514 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 474994711191=g '—‡N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48224561 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438741012615168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5307964467539 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5315716902912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445599494962176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚9411.844851916 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438741380706304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6858482347008 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48224561 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1624661368193024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30556091 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1629267131828563 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1624661368193024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175075 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48940527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14499544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48950514 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629978430492672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14499549 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1623959141933056 trainer/global_step‚5649 _step‚464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5317062299648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629978430492672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629977058922496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48950514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5306619867105 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627ÐcûÁ42˜- _wandbruntime‚9411  _runtime‚9411“Ë™‚ 3loss/SRqqt1/assignment_loss‚0.47298571390688887 2loss/FBt2/assignment_loss‚0.004865454564354949 1loss/SRqqt2/detection_loss‚0.9651391833580353 %loss/total_loss‚8.558792784236402 epoch‚14 _runtime‚9411.927923168 _step‚465 0loss/FRt1/assignment_loss‚1.0340260907969117 /loss/FRt2/detection_loss‚0.5270485575362717 /loss/FBt1/detection_loss‚0.3659338189551112 trainer/global_step‚5649 # _timestamp‚1.78432915855349e+09 0loss/FRt2/assignment_loss‚1.7474117249492271 2loss/SRqqt2/assignment_loss‚0.6165806563112842 .loss/FRt1/detection_loss‚1.652741303133179 1loss/SRqqt1/detection_loss‚0.6346332463991448 /loss/FBt2/detection_loss‚0.5146940575482754 2loss/FBt1/assignment_loss‚0.022732976777719287»²aJ š‡ _runtime‚9411.927923168 _step‚465 0loss/FRt1/assignment_loss‚1.0340260907969117 /loss/FRt2/detection_loss‚0.5270485575362717 /loss/FBt1/detection_loss‚0.3659338189551112 trainer/global_step‚5649 # _timestamp‚1.78432915855349e+09 0loss/FRt2/assignment_loss‚1.7474117249492271 2loss/SRqqt2/assignment_loss‚0.6165806563112842 .loss/FRt1/detection_loss‚1.652741303133179 1loss/SRqqt1/detection_loss‚0.6346332463991448 /loss/FBt2/detection_loss‚0.5146940575482754 2loss/FBt1/assignment_loss‚0.022732976777719287 3loss/SRqqt1/assignment_loss‚0.47298571390688887 2loss/FBt2/assignment_loss‚0.004865454564354949 1loss/SRqqt2/detection_loss‚0.9651391833580353 %loss/total_loss‚8.558792784236402 epoch‚14Ñ°Ä’–Ÿ›‚(:” ÎçêÒå‹­ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ fö;œ‚(:° ÎçêÒ„˜Ê®* disk./.usagePercent‚38.987859211339504$ disk./.usageGB‚580.9472770690918 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375" disk.nvme0c0n1.in‚ 20004.203125 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚74279.1005859375 disk.sda.out‚ 490.82421875% disk.nvme0n1.out‚74279.1005859375 disk.rbd1.in‚ 506.52734375* proc.memory.percent‚18.045878410339355# memory_percent‚82.2047472000122# proc.memory.rssMB‚ 2956.63671875 disk.rbd2.out‚58.78564453125 disk.dm-1.out‚ 490.82421875 cpu‚25.586049844314076 disk.sda2.in‚0 network.sent‚ 1.0703504e+07" disk.dm-0.out‚74279.1005859375 disk.nvme0n1.in‚ 20004.203125) proc.memory.availableMB‚ 2915.57421875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 20004.203125 disk.rbd2.in‚0.015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 490.82421875# disk.rbd1.out‚19355.68505859375 network.recv‚ 2.416714e+06ä´£ž‚(:“ ÝçêÒ…½”¬ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚328.649' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.72980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓØ2ž‚(:§ ÝçêÒñÍ®# disk.nvme0c0n1.out‚ 74288.796875 cpu‚25.586309664680858 disk.rbd1.in‚ 506.52734375 disk.sda3.out‚ 491.1328125) disk./.usagePercent‚38.98786586753539 disk.dm-1.in‚ 5.93359375$ disk.nvme0c0n1.in‚20005.16015625# proc.memory.rssMB‚ 2956.63671875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0# disk.rbd1.out‚19355.68505859375 disk.dm-0.in‚20005.16015625 disk.dm-0.out‚ 74288.796875" disk.nvme0n1.in‚20005.16015625 disk.sda.in‚ 5.93359375 disk.sda.out‚ 491.1328125 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 disk.drbd2165.out‚0! disk.nvme0n1.out‚ 74288.796875$ disk./.usageGB‚580.9473762512207 disk.sda3.in‚ 5.93359375$ memory_percent‚82.20388889312744* proc.memory.percent‚18.045878410339355 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.dm-1.out‚ 491.1328125 network.recv‚ 2.421964e+06 disk.sda1.in‚0) proc.memory.availableMB‚ 2915.71484375 network.sent‚ 1.0766635e+07 disk.rbd0.in‚0 disk.drbd2165.in‚09û˜žŸ‚(:“ ìçêÒݵԬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·œ—,8 ‚(:­ ìçêÒÕƒë® disk.sda3.out‚ 493.328125! disk.nvme0n1.in‚ 20005.9921875# disk.rbd1.out‚19355.74755859375 disk.sda1.in‚0# proc.memory.rssMB‚ 2956.63671875* proc.memory.percent‚18.045878410339355 network.recv‚ 2.425314e+06% disk.nvme0n1.out‚74307.5087890625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0) disk./.usagePercent‚38.98818254885525# disk.nvme0c0n1.in‚ 20005.9921875 disk.sda.in‚ 5.93359375) proc.memory.availableMB‚ 2915.46484375$ disk./.usageGB‚580.9520950317383 disk.sda.out‚ 493.328125 cpu‚25.586541272745382 disk.dm-1.in‚ 5.93359375' disk.nvme0c0n1.out‚74307.5087890625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚ 20005.9921875$ memory_percent‚82.20541477203369 network.sent‚ 1.0771714e+07 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚ 5.93359375" disk.dm-0.out‚74307.5087890625 disk.drbd2165.out‚0 disk.dm-1.out‚ 493.328125 disk.rbd1.in‚ 506.52734375+€ˆ9Ÿ¡‚(:” ûçêÒŸþì­ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÖ¢‚(:« ûçêÒ푦®$ disk./.usageGB‚581.0310554504395 disk.rbd0.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚74315.9228515625) proc.memory.availableMB‚ 2915.69921875$ disk.nvme0c0n1.in‚20088.07421875 disk.sda.in‚ 5.93359375 disk.sda2.in‚0 cpu‚25.58674983928821 network.sent‚ 1.077682e+07 disk.dm-1.in‚ 5.93359375" disk.nvme0n1.in‚20088.07421875# disk.rbd1.out‚19355.74755859375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚74315.9228515625 disk.sda2.out‚0* proc.memory.percent‚18.045878410339355 disk.rbd0.out‚0 disk.rbd2.in‚0.015625 disk.sda3.in‚ 5.93359375 disk.sda.ouÉvr²§t‚ 493.65625$ memory_percent‚82.20398426055908 proc.cpu.threads‚17 network.recv‚ 2.428869e+06 disk.dm-0.in‚20088.07421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 493.65625 disk.rbd1.in‚ 506.52734375 disk.rbd2.out‚58.78564453125" disk.dm-0.out‚74315.9228515625 disk.dm-1.out‚ 493.65625# proc.memory.rssMB‚ 2956.63671875) disk./.usagePercent‚38.993481648806498|Ô£‚(:‰ ŠèêÒ®”Ƭ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚249.397' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Llfé-¤‚(:¢ ŠèêÒÀü„¯ disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 494.296875# proc.memory.rssMB‚ 2956.63671875 disk.sda1.in‚0 disk.dm-1.out‚ 494.296875 disk.rbd1.in‚ 506.52734375& disk.nvme0c0n1.out‚74323.529296875 disk.rbd2.in‚0.015625 disk.nvme0n1.in‚ 20089.03125$ disk.nvme0n1.out‚74323.529296875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375 network.sent‚ 1.0781821e+07) disk./.usagePercent‚38.98818408490045 disk.sda.in‚ 5.93359375! disk.dm-0.out‚74323.529296875 disk.sda1.out‚0) proc.memory.availableMB‚ 2915.65234375 disk.sda2.out‚0 disk.sda3.in‚ 5.93359375 cpu‚25.587055884745013 disk.dm-0.in‚ 20089.03125# disk.rbd1.out‚19355.82568359375 network.recv‚ 2.431713e+06$ disk./.usageGB‚580.9521179199219! disk.nvme0c0n1.in‚ 20089.03125 disk.sda.out‚ 494.296875* proc.memory.percent‚18.045878410339355 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 disk.drbd2165.in‚0" memory_percent‚82.204270362854‰åwÇ2¥- _wandbruntime‚9490  _runtime‚9490«…9Ŧ¿ _runtime‚9490.686780617 _step‚466 % lr-AdamW/pg1‚0.000227539501845999 % lr-AdamW/pg2‚0.000227539501845999 trainer/global_step‚5699 % _timestamp‚1.7843292373099701e+09 pC­Ê§Ä trainer/global_step‚5699 % _timestamp‚1.7843292373099701e+09 _runtime‚9490.686780617 _step‚466 % lr-AdamW/pg1‚0.000227539501845999 % lr-AdamW/pg2‚0.000227539501845999Ò ;¹2¨- _wandbruntime‚9490  _runtime‚9490ÉA»êÿ'©ùO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1642927282577647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843292373135023e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442537846591488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49367774 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49357801 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6917671612416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1643644485593088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994429 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48626655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48626649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6917706148864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5351933513967 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1637575349063680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚5699 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30816899 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9490.688346994 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1638283377795072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1638283377795072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361107798016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1637575374491648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1638283403968512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1638283403968512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449455518203904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442538214682624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48626649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚467 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362453187584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1643644485593088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16192492 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1642928652605937 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49367774\žoÏ(ªþO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1643644485593088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994429 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48626655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48626649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6917706148864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5351933513967 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1637575349063680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚5699 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30816899 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9490.688346994 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1638283377795072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1638283377795072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361107798016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1637575374491648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1638283403968512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1638283403968512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449455518203904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442538214682624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48626649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚467 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362453187584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1643644485593088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16192492 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1642928652605937 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49367774 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1642927282577647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843292373135023e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442537846591488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49367774 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49357801 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6917671612416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49376266 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49366279 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48634855 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚9492.082491921 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011127 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚442615332999168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6918901217280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1643207457850471 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30822178 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5362032833536 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1638561404698624 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643206087822069 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚468 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354201580647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1637853230841856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011127 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49366279 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98001134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14626694 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5363378230272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30821929 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48634861 % _timestamp‚1.7843292387075782e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_endœ¹”æ/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640Í"bÈ'­ˆN GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1643207457850471 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30822178 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5362032833536 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1638561404698624 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643206087822069 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚468 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354201580647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1637853230841856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011127 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49366279 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98001134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14626694 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5363378230272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30821929 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48634861 % _timestamp‚1.7843292387075782e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1643924782928896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48634855 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5363378230272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14626699 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449534602307584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48634861 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1643923411358720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49376266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16195479 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16195235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49376266 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49366279 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48634855 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚9492.082491921 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011127 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚442615332999168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6918901217280ÔCèÉ2®- _wandbruntime‚9492  _runtime‚9492Û:Í–¯ý $loss/total_loss‚9.46734313257413 trainer/global_step‚5699 _step‚469 2loss/SRqqt1/assignment_loss‚0.5744618211276927 1loss/SRqqt2/assignment_loss‚0.615797950697655 1loss/FBt1/assignment_loss‚0.02522874668478982 1loss/FBt2/assignment_loss‚0.15203057760896707 /loss/FBt1/detection_loss‚0.4692890632956643 _runtime‚9492.165207238 ,loss/FRt1/detection_loss‚1.5715653504691 /loss/FRt2/detection_loss‚0.5358825092516679 1loss/SRqqt2/detection_loss‚0.9728931534568194 /loss/FBt2/detection_loss‚0.9351765057714335 % _timestamp‚1.7843292387907383e+09 0loss/FRt1/assignment_loss‚1.0907901349493647 0loss/FRt2/assignment_loss‚1.8946771278238608 1loss/SRqqt1/detection_loss‚0.6295501914371153 epoch‚14nÌ°‚ $loss/total_loss‚9.46734313257413 trainer/global_step‚5699 _step‚469 2loss/SRqqt1/assignment_loss‚0.5744618211276927 1loss/SRqqt2/assignment_loss‚0.615797950697655 1loss/FBt1/assignment_loss‚0.02522874668478982 1loss/FBt2/assignment_loss‚0.15203057760896707 /loss/FBt1/detection_loss‚0.4692890632956643 _runtime‚9492.165207238 ,loss/FRt1/detection_loss‚1.5715653504691 /loss/FRt2/detection_loss‚0.5358825092516679 1loss/SRqqt2/detection_loss‚0.9728931534568194 /loss/FBt2/detection_loss‚0.9351765057714335 % _timestamp‚1.7843292387907383e+09 0loss/FRt1/assignment_loss‚1.0907901349493647 0loss/FRt2/assignment_loss‚1.8946771278238608 1loss/SRqqt1/detection_loss‚0.6295501914371153 epoch‚14Õ“èÆ!”±‚(:‰ ™èêÒ´áÔ¯ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ@£!5²‚(:ª ™èêÒ‹°* disk./.usagePercent‚38.987974670737444$ disk./.usageGB‚580.9489974975586 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19355.82568359375 disk.rbd2.in‚0.015625 disk.rbd2.out‚58.78564453125 disk.sda1.out‚0 disk.nvme0n1.in‚ 20089.03125 disk.rbd0.out‚0 disk.dm-1.in‚ 5.93359375 disk.sda.in‚ 5.93359375 network.recv‚ 2.435248e+06 disk.sda2.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 20089.03125 network.sent‚ 1.0786187e+07 disk.drbd2165.out‚0 disk.dm-1.out‚ 495.296875( disk.nvme0c0n1.out‚74331.77880859375# proc.memory.rssMB‚ 2956.63671875* proc.memory.percent‚18.045878410339355 disk.sda2.out‚0# disk.dm-0.out‚74331.77880859375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 495.296875& disk.nvme0n1.out‚74331.77880859375$ memory_percent‚82.20295906066895( proc.memory.availableMB‚ 2915.8671875 disk.dm-0.in‚ 20089.03125 disk.sda.out‚ 495.296875 cpu‚25.587149206317303 proc.cpu.threads‚17Ÿ¹²pž³‚(:“ ¨èêÒ¶õþ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚254.229' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.84580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eà|1´‚(:Ž ¨èêÒè•÷­ disk.nvme0n1.out‚74343 disk.sda.in‚ 5.93359375 disk.dm-1.in‚ 5.93359375 network.sent‚ 1.0849358e+07 disk.drbd2165.in‚0 disk.dm-1.out‚ 496.890625* proc.memory.percent‚18.045878410339355 proc.cpu.threads‚17) disk./.usagePercent‚38.98797287868471$ disk.nvme0c0n1.in‚20089.86328125" disk.nvme0n1.in‚20089.86328125 disk.sda1.out‚0# proc.memory.rssMB‚ 2956.63671875 network.recv‚ 2.440523e+06 disk.sda.out‚ 496.890625$ memory_percent‚82.20429420471191 disk.nvme0c0n1.out‚74343 disk.sda2.out‚0 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19355.88818359375 disk.sda1.in‚0( proc.memory.availableMB‚ 2915.6484375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 cpu‚25.587391273944053 disk.sda3.in‚ 5.93359375 disk.sda2.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚580.9489707946777 disk.sda3.out‚ 496.890625 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-0.in‚20089.86328125 disk.dm-0.out‚74343ÿ}„ ”µ‚(:‰ ·èêÒÅÓÚ¬ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚172.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­i€=¶‚(:² ·èêÒ…µ˜¯ disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 497.296875& disk.nvme0n1.out‚74356.58935546875 disk.sda.in‚ 5.93359375$ memory_percent‚82.20434188842773 disk.rbd2.in‚0.015625 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚20208.15234375' proc.memory.availableMB‚ 2915.640625 cpu‚25.587587239877582 network.recv‚ 2.444392e+06 disk.dm-0.in‚20208.15234375# disk.rbd1.out‚19355.88818359375 network.sent‚ 1.0854998e+07* disk./.usagePercent‚38.988414491681276 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.045878410339355 proc.cpu.threads‚17# disk.dm-0.out‚74356.58935546875 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 497.296875( disk.nvme0c0n1.out‚74356.58935546875$ disk./.usageGB‚580.9555511474609 disk.rbd2.out‚58.78564453125" disk.nvme0n1.in‚20208.15234375 disk.rbd1.in‚ 506.52734375# proc.memory.rssMB‚ 2956.63671875 disk.rbd0.out‚0 disk.sda.out‚ 497.296875 disk.sda2.out‚0 disk.sda3.in‚ 5.93359375T¯ ·‚(:’ ÆèêÒƒò˜¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡¿5¯<¸‚(:± ÆèêҦ̭® disk.sda3.in‚ 5.93359375 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd2.in‚0.015625 disk.dm-0.in‚20208.15234375 disk.drbd2165.in‚0$ memory_percent‚82.20336437225342 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚18.045878410339355 network.recv‚ 2.447806e+06# disk./.usageGB‚580.955696105957 disk.sda.out‚ 498.08984375# proc.memory.rssMB‚ 2956.63671875 proc.cpu.threads‚17 network.sent‚ 1.0859542e+07 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 498.08984375& disk.nvme0c0n1.out‚74363.916015625# disk.rbd1.out‚19355.95068359375 disk.rbd0.in‚0 disk.sda3.out‚ 498.08984375$ disk.nvme0c0n1.in‚20208.15234375$ disk.nvme0n1.out‚74363.916015625) disk./.usagePercent‚38.98842421996757" disk.nvme0n1.in‚20208.15234375 disk.sda2.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚74363.916015625 cpu‚25.58776821263169 disk.sda.in‚ 5.93359375) proc.memory.availableMB‚ 2915.80078125þ™Z “¹‚(:ˆ ÕèêÒ½Ýà« gpu.0.gpu‚84.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚343.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.741 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!µñ21º‚(:¦ ÕèêÒÏàÆ® disk.sda2.in‚0" disk.dm-0.out‚74373.7587890625' disk.nvme0c0n1.out‚74373.7587890625 disk.sda2.out‚0 disk.drbd2165.in‚0* disk./.usagePercent‚38.988446492623055 disk.rbd2.out‚58.78564453125 disk.nvme0n1.in‚ 20208.984375) proc.memory.availableMB‚ 2915.26953125 network.recv‚ 2.451161e+06 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 20208.984375# memory_percent‚82.2066068649292# proc.memory.rssMB‚ 2956.63671875 disk.dm-0.in‚ 20208.984375 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 498.40625% disk.nvme0n1.out‚74373.7587890625 disk.sda.in‚ 5.93359375 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 cpu‚25.58807333625684$ disk./.usageGB‚580.9560279846191 disk.sda.out‚ 498.40625# disk.rbd1.out‚19356.02880859375 disk.sda1.out‚0 disk.dm-1.in‚ 5.93359375 proc.cpu.threads‚17 network.sent‚ 1.0864606e+07 disk.rbd0.out‚0 disk.dm-1.out‚ 498.40625* proc.memory.percent‚18.045878410339355ý\"»‚(:— äèêÒéÂÈ­$ disk.nvme0c0n1.in‚20209.94140625 disk.nvme0n1.out‚ 74381.8125 disk.rbd2.in‚0.015625 disk.dm-0.out‚ 74381.8125# proc.memory.rssMB‚ 2956.63671875* proc.memory.percent‚18.045878410339355 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.dm-1.out‚498.8125 cpu‚25.588227845178782 disk.sda.out‚498.8125 disk.rbd2.out‚58.78564453125) proc.memory.availableMB‚ 2915.23046875 network.recv‚ 2.454361e+06" disk.nvme0n1.in‚20209.94140625 disk.sda2.out‚0# disk.rbd1.out‚19356.02880859375 disk.dm-0.in‚20209.94140625 disk.drbd2165.out‚0# memory_percent‚82.2068452835083 disk.sda3.in‚ 5.93359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 5.93359375$ disk./.usageGB‚580.9560623168945! disk.nvme0c0n1.out‚ 74381.8125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.out‚498.8125 disk.sda.in‚ 5.93359375 disk.sda2.in‚0 network.sent‚ 1.0870213e+07) disk./.usagePercent‚38.98844879669086({””¼‚(:‰ äèêҘ뿭 gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#{‰¥2½- _wandbruntime‚9570  _runtime‚9570_@Síɾà ' lr-AdamW/pg1‚0.00022671200230297848 ' lr-AdamW/pg2‚0.00022671200230297848 trainer/global_step‚5749 % _timestamp‚1.7843293174597929e+09 _runtime‚9570.834355753 _step‚470ð‹ÂÎ¿È ' lr-AdamW/pg2‚0.00022671200230297848 trainer/global_step‚5749 % _timestamp‚1.7843293174597929e+09 _runtime‚9570.834355753 _step‚470 ' lr-AdamW/pg1‚0.00022671200230297848ÖP–¦2À- _wandbruntime‚9570  _runtime‚9570µ. åÿ'ÁùO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453389760776192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49036955 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1657590838029312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31082032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16330461 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6977630434304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1652183414300672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5407423728640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5407423728640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14751301 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5399515227397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31081762 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49783553 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9570.835527485 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14751306 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6977593800704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1656868978627845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚471 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5408769118208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49793526 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830481 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1656867608599555 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843293174605656e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1652183414300672 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49036955 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49783553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1657592209592320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1657590838029312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚453390165500928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446412166975488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651469437972480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1657592209592320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5398170627075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49036949 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1652183440474112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49036949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16330726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175IŽ×rÂþO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16330726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453389760776192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49036955 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1657590838029312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31082032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?6nð}t#9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16330461 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6977630434304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1652183414300672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5407423728640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5407423728640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14751301 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5399515227397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31081762 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49783553 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9570.835527485 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14751306 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6977593800704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1656868978627845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚471 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5408769118208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49793526 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830481 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1656867608599555 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843293174605656e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1652183414300672 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49036955 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49783553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1657592209592320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1657590838029312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚453390165500928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446412166975488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651469437972480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1657592209592320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5398170627075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49036949 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1652183440474112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49036949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008ׄöAÈ2Ã- _wandbruntime‚9572  _runtime‚9572 å+ 'ăN HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14753844 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31087300 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847179 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837186 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16333198 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446490021474304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 trainer/global_step‚5749 _runtime‚9572.228054593 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1652461415030784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49802018 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446489653383168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1651747345178624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1657869763794944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49045155 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1652461415030784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837186 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31087042 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847179 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843293188532522e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49802018 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5408348764160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453468838252544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5408348764160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49045161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5409694160896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49045155 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14753844 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31087300 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847179 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837186 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16333198 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446490021474304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 trainer/global_step‚5749 _runtime‚9572.228054593 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1652461415030784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49802018 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446489653383168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1651747345178624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1657869763794944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49045155 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1652461415030784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837186 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31087042 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847179 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843293188532522e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49802018 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5408348764160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453468838252544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5408348764160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49045161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5409694160896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49045155 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 B΂(:³ ‘éêÒÈÎή disk.sda3.in‚ 5.93359375$ disk.nvme0c0n1.in‚20323.70703125# disk.rbd1.out‚19356.18505859375 disk.sda1.out‚0% disk.nvme0n1.out‚74406.6357421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2956.63671875 proc.cpu.threads‚17" disk.dm-0.out‚74406.6357421875 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375$ disk./.usageGB‚580.9522666931152 disk.dm-0.in‚20323.70703125 disk.sda2.in‚0 disk.rbd2.in‚0.015625' disk.nvme0c0n1.out‚74406.6357421875 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.rbd0.out‚0 network.sent‚ 1.0943288e+07 network.recv‚ 2.466915e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd2.out‚58.78564453125# memory_percent‚88.1366491317749) proc.memory.availableMB‚ 1943.69140625* proc.memory.percent‚18.045878410339355 disk.dm-1.out‚ 499.8515625 disk.sda3.out‚ 499.8515625 disk.sda.out‚ 499.8515625 disk.rbd1.in‚ 506.52734375 cpu‚25.588743699583997* disk./.usagePercent‚38.988194069194286" disk.nvme0n1.in‚20323.70703125?H¿’Ï‚(:‡  éêÒΑº¬ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚341.826' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.3652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#cûCЂ(:¸  éêÒ€²Ã­# disk.rbd1.out‚19356.18505859375 disk.sda3.out‚ 500.08203125& disk.nvme0n1.out‚74410.11669921875 disk.sda.out‚ 500.08203125 cpu‚25.589026176004925 proc.cpu.threads‚17 network.recv‚ 2.470069e+06 disk.dm-0.in‚20323.70703125 disk.sda2.in‚0$ memory_percent‚88.13652992248535 disk.rbd2.in‚0.015625 disk.dm-1.out‚ 500.08203125$ disk.nvme0c0n1.in‚20323.70703125( disk.nvme0c0n1.out‚74410.11669921875" disk.nvme0n1.in‚20323.70703125 disk.rbd0.in‚0* proc.memory.percent‚18.045878410339355) disk./.usagePercent‚38.98780698580252 disk.sda.in‚ 5.93359375 disk.sda2.out‚0( proc.memory.availableMB‚ 1943.7109375 network.sent‚ 1.0948352e+07 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚74410.11669921875 disk.dm-1.in‚ 5.93359375 disk.sda3.in‚ 5.93359375# proc.memory.rssMB‚ 2956.63671875$ disk./.usageGB‚580.9464988708496 disk.drbd2165.in‚0 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0‡¸s„”Ñ‚(:‰ ¯éêÒ§†æ« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.597' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×%Œ¬=Ò‚(:² ¯éêÒ¬¡ý¯ disk.rbd1.in‚ 506.52734375 disk.rbd0.in‚0" disk.dm-0.out‚74419.8525390625 disk.sda1.out‚0( proc.memory.availableMB‚ 1943.9453125* proc.memory.percent‚18.045878410339355 disk.dm-1.out‚ 500.45703125 disk.sda3.out‚ 500.45703125$ memory_percent‚88.13509941101074 disk.sda3.in‚ 5.93359375% disk.nvme0n1.out‚74419.8525390625 disk.rbd0.out‚0 cpu‚25.589158915756443# disk.rbd1.out‚19356.24755859375' disk.nvme0c0n1.out‚74419.8525390625 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda2.in‚0 disk.dm-1.in‚ 5.93359375# proc.memory.rssMB‚ 2956.63671875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 20457.5390625 network.recv‚ 2.473653e+06) disk./.usagePercent‚38.98780877785526 disk.sda.in‚ 5.93359375 disk.sda.out‚ 500.45703125 disk.dm-0.in‚ 20457.5390625 disk.sda1.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.0953573e+07$ disk./.usageGB‚580.9465255737305 disk.rbd2.out‚58.78564453125# disk.nvme0c0n1.in‚ 20457.5390625 disk.sda2.out‚0¿Mò™2Ó- _wandbruntime‚9650  _runtime‚9650õ™OæÇÔÁ & lr-AdamW/pg1‚0.0002258787428583742 & lr-AdamW/pg2‚0.0002258787428583742 trainer/global_step‚5799 % _timestamp‚1.7843293969507902e+09 _runtime‚9650.325392608 _step‚474óxGÌÕÆ _step‚474 & lr-AdamW/pg1‚0.0002258787428583742 & lr-AdamW/pg2‚0.0002258787428583742 trainer/global_step‚5799 % _timestamp‚1.7843293969507902e+09 _runtime‚9650.325392608Ú#ÔYF2Ö- _wandbruntime‚9650  _runtime‚9650â ÝZÿ'×ùO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49447255 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1670807934621463 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450286487359488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50219278 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5453739659264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49447255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31347044 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1666083476979712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5453739659264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7037396844544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1665363552309248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50219278 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1666083450806272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666533 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14878456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457323884204032 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31346797 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50209305 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49447249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1665363526881280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1671538562028544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1666083450806272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5455085048832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450286855450624 % _timestamp‚1.7843293969519057e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1671537190465536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14878451 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1671538562028544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1670809304649753 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5444407740183 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1671537190465536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16468346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7037429283840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457324284734464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50209305 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175•ÀÀ™(ØþO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1666083450806272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666533 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14878456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457323884204032 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31346797 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50209305 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49447249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1665363526881280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1671538562028544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1666083450806272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5455085048832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450286855450624 % _timestamp‚1.7843293969519057e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1671537190465536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14878451 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1671538562028544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1670809304649753 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5444407740183 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1671537190465536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16468346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7037429283840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457324284734464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50209305 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49447255 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1670807934621463 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450286487359488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50219278 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5453739659264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49447255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31347044 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1666083476979712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5453739659264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7037396844544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1665363552309248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50219278Ûs´ï2Ù- _wandbruntime‚9651  _runtime‚9651Ç¡EãäÚƒN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16471088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1671088109894287 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚9651.716764129 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1666361451536384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1665641434087424 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1666361477709824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5446675806863 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843293983419194e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31352327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450363973767168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14880994 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7038632011776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5454664694784 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1671817487801344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 trainer/global_step‚5799 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5456010091520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50217783 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1671816116231168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450364341858304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673238 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49455455 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50227770 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1666361477709824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673238 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457402571249664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1665641408659456 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31352082 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5445331206429 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7038597482496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1671817487801344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457402973870080 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50217783 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1671816116231168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450364341858304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673238 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49455455 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50227770 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1666361477709824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673238 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457402571249664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1665641408659456 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31352082 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5445331206429 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7038597482496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1671817487801344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457402973870080 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16471088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1671088109894287 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚9651.716764129 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1666361451536384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1665641434087424 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1666361477709824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5446675806863 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843293983419194e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31352327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450363973767168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14880994 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7038632011776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5454664694784 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1671817487801344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 trainer/global_step‚5799 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5456010091520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 B2é- _wandbruntime‚9729  _runtime‚9729Œ³ÝÉêà ' lr-AdamW/pg1‚0.00022503977703088317 ' lr-AdamW/pg2‚0.00022503977703088317 trainer/global_step‚5849 % _timestamp‚1.7843294765091894e+09 _runtime‚9729.883831694 _step‚478 øîRÎëÈ ' lr-AdamW/pg2‚0.00022503977703088317 trainer/global_step‚5849 % _timestamp‚1.7843294765091894e+09 _runtime‚9729.883831694 _step‚478 ' lr-AdamW/pg1‚0.00022503977703088317Þ¤Èâá2ì- _wandbruntime‚9729  _runtime‚9729Cé+Ãû#íýO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50645030 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚454161175834624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461258195883008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1679983513485312 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49857549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1685483542901760 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15005606 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5501400979456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16607062 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31612668 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843294765104103e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5501400979456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚454160807743488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1684749630671661 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16606813 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461258598510592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7097422675968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50645030 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1685484914464768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49857549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7097388139520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5490644853291 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679983487311872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31612414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502585 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚479 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679983487311872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1685484914464768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1679257641218048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50635057 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1679257615790080 _runtime‚9729.885166244 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15005601 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5500055589888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49857555 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1684748260643371 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚5849 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5500055589888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492606ð+]ò(î‚P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15005601 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5500055589888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49857555 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1684748260643371 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚5849 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5500055589888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492606 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50645030 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚454161175834624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461258195883008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1679983513485312 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49857549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1685483542901760 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15005606 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5501400979456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16607062 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31612668 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843294765104103e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5501400979456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚454160807743488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1684749630671661 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16606813 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461258598510592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7097422675968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50645030 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1685484914464768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49857549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7097388139520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5490644853291 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679983487311872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31612414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502585 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚479 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679983487311872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1685484914464768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1679257641218048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50635057 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1679257615790080 _runtime‚9729.885166244ßSº¾2ï- _wandbruntime‚9731  _runtime‚9731$¦)i 'ð‡N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _runtime‚9731.284480714 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31617719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16609575 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7098584452096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49865761 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50653522 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5492912919971 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5491568319537 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1680261488041984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1680261514215424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49865761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚5849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16609816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461337281223680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1679535497568256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843294779095364e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5500980625408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1680261514215424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509290 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5502326022144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1685762468667392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31617965 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49865755 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461336878603264 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1685027065887793 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚480 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50643535 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519283 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1685028435916195 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1685763840237568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚454238662242304 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31617719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16609575 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7098584452096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49865761 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50653522 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5492912919971 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5491568319537 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1680261488041984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1680261514215424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49865761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚5849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16609816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461337281223680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1679535497568256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843294779095364e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5500980625408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1680261514215424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509290 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5502326022144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1685762468667392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31617965 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49865755 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461336878603264 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1685027065887793 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚480 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50643535 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519283 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1685028435916195 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1685763840237568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚454238662242304 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 Fö‚(:³ ‰êêÒۈ̯ disk.sda.out‚ 504.33203125 disk.rbd2.out‚58.78564453125 disk.sda1.in‚0 disk.dm-1.out‚ 504.33203125* proc.memory.percent‚18.045878410339355 proc.cpu.threads‚17 disk.rbd2.in‚0.015625 disk.sda1.out‚0 disk.sda3.in‚ 5.93359375& disk.nvme0n1.out‚74475.84130859375 cpu‚25.590490668739267 disk.sda2.in‚0 disk.rbd0.in‚0! disk.nvme0n1.in‚ 20568.5078125 disk.dm-0.in‚ 20568.5078125 disk.drbd2165.out‚0& proc.memory.availableMB‚ 1943.90625 network.recv‚ 2.495292e+06$ disk./.usageGB‚580.9543800354004 disk.rbd1.in‚ 506.52734375 disk.rbd0.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 20568.5078125) disk./.usagePercent‚38.98833589736826# disk.rbd1.out‚19356.59130859375# proc.memory.rssMB‚ 2956.63671875 network.sent‚ 1.1042722e+07 disk.sda.in‚ 5.93359375 disk.sda2.out‚0$ memory_percent‚88.13533782958984# disk.dm-0.out‚74475.84130859375 disk.dm-1.in‚ 5.93359375 disk.sda3.out‚ 504.33203125( disk.nvme0c0n1.out‚74475.84130859375¬kR_”÷‚(:‰ ˜êêÒ…¶ã« gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,UÇ;ø‚(:° ˜êêÒàšÈ® disk.sda3.in‚ 5.93359375 disk.sda.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375# disk.rbd1.out‚19356.59130859375! disk.dm-0.out‚74479.306640625! disk.nvme0n1.in‚ 20568.5078125$ disk.nvme0n1.out‚74479.306640625 disk.sda.out‚ 504.79296875# proc.memory.rssMB‚ 2956.63671875 disk.sda2.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1944.05859375* proc.memory.percent‚18.045878410339355$ disk./.usageGB‚580.9543304443359# disk.nvme0c0n1.in‚ 20568.5078125 disk.rbd2.in‚0.015625 proc.cpu.threads‚17 disk.drbd2165.in‚0 cpu‚25.590664207710507 network.recv‚ 2.500375e+06) disk./.usagePercent‚38.98833256927031 disk.dm-0.in‚ 20568.5078125 disk.sda1.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 504.79296875 disk.sda3.out‚ 504.79296875& disk.nvme0c0n1.out‚74479.306640625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚58.78564453125$ memory_percent‚88.13440799713135 network.sent‚ 1.1105345e+07 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375¬8zŸù‚(:” §êêÒ›™Å¬ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!êCé4ú‚(:© §êêÒÑÐÓ® disk.sda3.in‚ 5.93359375( disk.nvme0c0n1.out‚74489.47412109375 network.sent‚ 1.111042e+07# disk.dm-0.out‚74489.47412109375 disk.rbd0.out‚0 disk.drbd2165.in‚0* disk./.usagePercent‚38.988342809571684 disk.sda.out‚ 505.28125 disk.sda1.in‚0 disk.dm-1.out‚ 505.28125) proc.memory.availableMB‚ 1943.84765625 disk.nvme0n1.in‚ 20689.140625 disk.sda3.out‚ 505.28125# memory_percent‚88.1356954574585 cpu‚25.590907829122685 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 20689.140625* proc.memory.percent‚18.045878410339355 network.recv‚ 2.503724e+06 disk.sda.in‚ 5.93359375# disk.rbd1.out‚19356.65380859375 disk.dm-0.in‚ 20689.140625# proc.memory.rssMB‚ 2956.63671875$ disk./.usageGB‚580.9544830322266 disk.sda2.out‚0 disk.rbd2.out‚58.78564453125 disk.dm-1.in‚ 5.93359375& disk.nvme0n1.out‚74489.47412109375 disk.sda2.in‚0 disk.rbd1.in‚ 506.52734375 disk.drbd2165.out‚0‘)~”û‚(:‰ ¶êêÒŸ Ë¬ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚207.692' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ 4Aü‚(:¶ ¶êêÒø›¡¯ disk.rbd0.in‚0 disk.rbd2.in‚0.015625 disk.sda3.out‚ 505.54296875 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 5.93359375$ disk.nvme0c0n1.in‚20689.97265625" disk.nvme0n1.in‚20689.97265625 network.sent‚ 1.1115525e+07 disk.sda2.out‚0# disk.rbd1.out‚19356.65380859375 disk.dm-1.out‚ 505.54296875& disk.nvme0n1.out‚74497.12451171875* proc.memory.percent‚18.045878410339355 disk.sda2.in‚0 disk.rbd2.out‚58.78564453125# disk.dm-0.out‚74497.12451171875( disk.nvme0c0n1.out‚74497.12451171875$ memory_percent‚88.13791275024414 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚20689.97265625 disk.sda3.in‚ 5.93359375# disk./.usageGB‚580.954460144043 disk.sda.out‚ 505.54296875 cpu‚25.591173410631345 disk.sda.in‚ 5.93359375 disk.drbd2165.in‚0 network.recv‚ 2.506951e+06) disk./.usagePercent‚38.98834127352648' proc.memory.availableMB‚ 1943.484375# proc.memory.rssMB‚ 2956.63671875šØ”ý‚(:‰ ÅêêÒæé·¬ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.148' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.4296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Fº=þ‚(:² Åêê҃⮠network.recv‚ 2.510579e+06" disk.nvme0n1.in‚20689.97265625 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1943.33984375 cpu‚25.59083314551573 proc.cpu.threads‚17 disk.sda.in‚ 5.93359375 disk.rbd2.in‚0.015625 disk.rbd0.out‚0 disk.dm-0.in‚20689.97265625 disk.sda1.out‚0* proc.memory.percent‚18.045878410339355$ disk.nvme0n1.out‚74504.779296875 disk.rbd0.in‚0# disk.rbd1.out‚19356.73193359375$ memory_percent‚88.13879489898682 network.sent‚ 1.1120591e+07 disk.sda.out‚ 505.93359375 disk.rbd2.out‚58.78564453125 disk.dm-1.in‚ 5.93359375# proc.memory.rssMB‚ 2956.63671875) disk./.usagePercent‚38.98833896945867$ disk./.usageGB‚580.9544258117676 disk.rbd1.in‚ 506.52734375 disk.sda1.in‚0 disk.dm-1.out‚ 505.93359375! disk.dm-0.out‚74504.779296875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 5.93359375 disk.sda3.out‚ 505.93359375$ disk.nvme0c0n1.in‚20689.97265625& disk.nvme0c0n1.out‚74504.779296875Ùa™žÿ‚(:“ ÔêêÒá©Ô« gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¢.@€‚(:µ ÔêêÒèøÒ® network.sent‚ 1.1125661e+07 network.recv‚ 2.513666e+06 disk.sda3.in‚ 5.93359375 disk.rbd1.in‚ 506.52734375 disk.rbd2.out‚58.78564453125 disk.drbd2165.out‚0$ disk./.usageGB‚580.9544563293457( disk.nvme0c0n1.out‚74514.41357421875! disk.nvme0n1.in‚ 20690.9296875# disk.rbd1.out‚19356.73193359375 disk.rbd2.in‚0.015625# disk.dm-0.out‚74514.41357421875) proc.memory.availableMB‚ 1942.80078125* proc.memory.percent‚18.045878410339355 disk.dm-0.in‚ 20690.9296875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚74514.41357421875 disk.drbd2165.in‚0 cpu‚25.59051732101615 disk.sda.out‚ 506.21484375 disk.rbd0.in‚0$ memory_percent‚88.14208507537842 proc.cpu.threads‚17) disk./.usagePercent‚38.98834101751895 disk.sda3.out‚ 506.21484375# disk.nvme0c0n1.in‚ 20690.9296875 disk.sda1.out‚0# proc.memory.rssMB‚ 2956.63671875 disk.sda.in‚ 5.93359375 disk.sda2.out‚0 disk.dm-1.in‚ 5.93359375 disk.dm-1.out‚ 506.21484375Ë:2- _wandbruntime‚9810  _runtime‚9810k/…ŠÉ‚à ' lr-AdamW/pg1‚0.00022419515870571277 ' lr-AdamW/pg2‚0.00022419515870571277 trainer/global_step‚5899 % _timestamp‚1.7843295570096867e+09 _runtime‚9810.384432028 _step‚482¾¬¤ ÎƒÈ ' lr-AdamW/pg1‚0.00022419515870571277 ' lr-AdamW/pg2‚0.00022419515870571277 trainer/global_step‚5899 % _timestamp‚1.7843295570096867e+09 _runtime‚9810.384432028 _step‚482â_Ž å2„- _wandbruntime‚9810  _runtime‚9810ø3 ý(…ûO ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1693151704698880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51070782 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16745007 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚5899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458035128127488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693883523817472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31877763 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1698689956693569 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1699429895337984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51060809 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50267855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458035496218624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1699431266900992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465192844936192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50267849 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51070782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7157348717568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784329557010967e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1693883549990912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31877518 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1693883549990912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693883523817472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546371520512 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1699431266900992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465192444405760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1699429895337984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51060809 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15132751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1693151730126848 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328658 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50267855 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5547716910080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546371520512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16744767 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50267849 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5547716910080 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5538226566721 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5536881966399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465192844936192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50267849 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51070782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7157348717568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784329557010967e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1693883549990912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31877518 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1693883549990912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693883523817472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546371520512 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1699431266900992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465192444405760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1699429895337984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51060809 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15132751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1693151730126848 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328658 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50267855 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5547716910080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546371520512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16744767 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50267849 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5547716910080 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5538226566721 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5536881966399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1699429895337984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51060809 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50267855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458035496218624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1699431266900992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979ãûòÙ½2‡- _wandbruntime‚9811  _runtime‚9811cm¡ãY ˆ…N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458112982626304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5547296556032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7158556958720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7158522429440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51069287 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1698968761938103 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1699710192673792 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1694161524547584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465271539585024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15135299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1698967391909701 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.páÈa"²eak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31882819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50276061 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465271136964608 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355335 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694161550721024 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1699710192673792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1693429611905024 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355335 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51079274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694161550721024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5547296556032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50276061 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458112614535168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5548641952768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50276055 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚9811.783966779 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16747759˜4ø'‰ŠN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51069287 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16747525 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5537805432645 _step‚484 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31883058 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465271136964608 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355335 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694161550721024 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1699710192673792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1693429611905024 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355335 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51079274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694161550721024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5547296556032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50276061 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458112614535168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5548641952768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50276055 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚9811.783966779 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16747759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458112982626304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5547296556032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7158556958720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7158522429440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51069287 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1698968761938103 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1699710192673792 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1694161524547584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465271539585024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15135299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1698967391909701 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31882819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50276061 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713377619337216 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713376247774208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50678155 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51486561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1707783560323072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174689 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713376247774208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461909448511488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7217425787904 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1707045793607680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16883066 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1707045819035648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5592687451136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1712628912687187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50678155 _runtime‚ 9890.48241966 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16883310 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1707783560323072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5584463679829 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164710 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocatioàâó™–n.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1712630282715477 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5594032840704 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5583119079507 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32143216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50678149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51496534 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461909816602624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15259901 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5592687451136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1707783586496512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1707783586496512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5594032840704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51496534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469126839762944UÉJ«(œP ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1707783586496512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1707783586496512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5594032840704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51496534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469126839762944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713377619337216 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713376247774208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50678155 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51486561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1707783560323072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174689 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713376247774208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461909448511488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7217425787904 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1707045793607680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16883066 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1707045819035648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5592687451136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1712628912687187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50678155 _runtime‚ 9890.48241966 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16883310 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1707783560323072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5584463679829 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164710 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1712630282715477 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5594032840704 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5583119079507 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32143216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50678149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51496534 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461909816602624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15259901 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5592687451136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175ç˜k¬2- _wandbruntime‚9891  _runtime‚98912"tÛ 'ž‡N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181394 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50686361 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32148516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191387 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713656545110016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15262449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16885827 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5593612486656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50686355 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5594957883392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15262444 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5949 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32148271 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1708061561053184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5593612486656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461986934919168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7218594748416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7218629277696 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1708061587226624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1707323700813824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5584042545753 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1707323675385856 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713655173539840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1712909087960011 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843296385001438e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102181394 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51505026 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469205932288000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1708061587226624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚488 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0÷B£ŸŒN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713656545110016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9891.875017364 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181394 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50686361 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32148516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191387 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713656545110016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15262449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16885827 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5593612486656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50686355 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5594957883392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15262444 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5949 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32148271 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1708061561053184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5593612486656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461986934919168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7218594748416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7218629277696 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1708061587226624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1707323700813824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5584042545753 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1707323675385856 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713655173539840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1712909087960011 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843296385001438e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 1021813Î…¨) 94 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51505026 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469205932288000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1708061587226624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚488 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1727322600210432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1727322600210432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465784136986624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465783768895488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51088449 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000762 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1721683596828672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51088455 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1727323971773440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1720939907944448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5639003381760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51088449 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1721683623002112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32407983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1727323971773440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7277453207040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32407736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15387056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7277418670592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5640348771328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784329717614033e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51088455 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _runtime‚9970.988786019 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639003381760 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568#´TH(²P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473061187566080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚5999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51922286 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1721683623002112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17020685 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15387051 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473061590193664 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1727322600210432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1727322600210432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465784136986624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465783768895488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51088449 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000762 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1721683596828672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51088455 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1727323971773440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1720939907944448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5639003381760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51088449 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1721683623002112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32407983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1727323971773440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7277453207040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32407736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15387056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7277418670592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5640348771328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784329717614033e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51088455 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _runtime‚9970.988786019 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639003381760 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5640348771328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17020927ëu–¶2³- _wandbruntime‚9972  _runtime‚99728¶FÌ´‡N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_ Ô|âA&batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51096655 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027439 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32413243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17023644 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5641273814016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51096661 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚492 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1726849413981919 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027439 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473139872932352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51920791 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32413000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51096661 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5639928417280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1727602897546240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473140275552768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465861623394304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51096655 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1727601525976064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5641273814016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15389599 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017446 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1721217764294656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1721961597558784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1721961597558784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1726848043953517 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51920791 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0QÑ4'µŒN F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027439 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32413243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17023644 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5641273814016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51096661 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚492 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1726849413981919 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027439 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473139872932352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51920791 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32413000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51096661 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5639928417280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1727602897546240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473140275552768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465861623394304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51096655 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1727601525976064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5641273814016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15389599 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017446 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1721217764294656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1721961597558784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1721961597558784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1726848043953517 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51920791 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51096655 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 Q&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6404415965080261 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6667328476905823 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35541635751724243 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12163212150335312 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6052988767623901 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14268843829631805 _step‚494 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9881078600883484 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1425871104001999 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35212674736976624 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.777815580368042 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11213728040456772 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13447430729866028 6validation_average_jet_accuracy‚0.3350408971309662 epoch‚14 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5949017405509949 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 particle/accuracy_4_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.020959168672561646 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7147861123085022 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6920467615127563 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6382842063903809 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1484052985906601 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17249999940395355 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6524900794029236 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6115204691886902 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.643204391002655 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6627923250198364 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1364339292049408 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.46267169713974 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8082443475723267 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4678686261177063 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7540717720985413 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41150468587875366 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1484052985906601 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2861541211605072 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.042479656636714935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.71161985397