:W&Bá¾T›©ÍM llumpg4pª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0#€¼Sû‚" 8fh1u4sf9zjl llumpg4pŠ× llumpg4pjuju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚512  dataset_limit‚0.2 dropout‚0.028897472506508114 epochs‚10  hidden_dim‚256 ' learning_rate‚0.0010135864482545187  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚6  num_encoder_layers‚4Bjjtttt_blocks_llumpg4pbs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ “èÔ°Ú¬¢š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64_—ˆ7 llumpg4p&  _runtime‚0  _wandb.runtime‚0¾Œæ›jÒä -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 ’èÔÈÅÛø*-of*=/data/spatop/options_files/sweep_tttt_jj/config_llumpg4p.json*-l* /scratch/logs*-n*jjtttt_blocks_llumpg4p2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº i553j9dzv5bnzso6kiq9pgou1o9ljuam‰Á½èÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€àðà£$ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º i553j9dzv5bnzso6kiq9pgou1o9ljuamµª½Ç2  requirements.txt)æLW llumpg4pZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64fƽY llumpg4pZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64€“J*  llumpg4pb trainer/global_step2¾¿ªÌ\  llumpg4pZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64:Nc/  llumpg4pb*"trainer/global_step2ŠBÖK  llumpg4pj: ”èÔØ“É«*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] äŒ ’  llumpg4pj€  –èÔ¨Øñî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 5.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 214 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ 8¬üš[ llumpg4pjÉ –èÔ…¶·Trainable params: 223 M Non-trainable params: 86 Total params: 223 M Total estimated model params size (MB): 893.650 Modules in train mode: 5866 Modules in eval mode: 0 Total FLOPs: 0 ÷³=*p llumpg4pj_ –èÔ¨ôµPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). Üêîz llumpg4pji –èÔèáÐZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). vL%þx llumpg4pjg –èÔÚêXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ¬ø llumpg4pjn –èÔ€Ùú_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). LÒÂy llumpg4pjh –èÔÈš‡Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). õ‚  s llumpg4pjb –èÔ¸Ÿ•Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). Dic¶p llumpg4pj_ –èÔø¼ Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). †k =s llumpg4pjb –èÔÈŠ¶Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). hq m llumpg4pj\ –èÔèÿÃMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). 5íœl llumpg4pj[ –èÔÛLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ¿×AÜ llumpg4p*Ê  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"6LB‰ê llumpg4pjØ –èÔÈÌäÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ©Œm^M llumpg4pj» –èÔÀÖ§«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. hÏï ë llumpg4pjÙ œèÔðþ°ïÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. µ-oÒP llumpg4pj¾ œèÔ˜äúï­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ø·zËŽ‚(:„ £èÔÁ¶êà gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.02009582519531* gpu.0.memoryAllocatedBytes‚ 58428817408 gpu.0.temp‚32.0 gpu.0.powerWatts‚142.54' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.508 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BEµJ‚(:À £èÔó«‰å" disk.nvme0c0n1.out‚ 130.4140625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0# memory_percent‚90.8151388168335 proc.memory.rssMB‚ 3217.03125 proc.cpu.threads‚17 network.sent‚34441 network.recv‚39451) disk./.usagePercent‚77.91579243507158% disk./.usageGB‚1161.0016136169434 disk.sda.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 37.48046875 disk.nvme0n1.in‚ 37.48046875! disk.nvme0c0n1.in‚ 37.48046875 disk.sda.out‚ 0.75390625 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 0.75390625 cpu‚15.519483015612078 disk.sda1.in‚0 disk.dm-1.out‚ 0.75390625 disk.sda2.out‚0 disk.dm-0.out‚ 130.4140625 disk.nvme0n1.out‚ 130.4140625) proc.memory.availableMB‚ 1504.84765625) proc.memory.percent‚19.63520050048828‹;¡;[ ‚(:Ñ ²èÔÂ×·ß network.sent‚44562& disk.nvme0c0n1.out‚286.07080078125 disk.sda.in‚0 disk.drbd1260.out‚ 0.00390625 disk.dm-0.in‚ 37.484375! disk.dm-0.out‚286.07080078125 disk.nvme0n1.in‚ 37.484375 proc.cpu.threads‚17% disk./.usageGB‚1161.0406684875488 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1.10546875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 1.10546875! proc.memory.rssMB‚ 3218.109375 network.recv‚44225) disk./.usagePercent‚77.91841344020712 disk.sda.out‚ 1.10546875 disk.dm-1.in‚0$ disk.nvme0n1.out‚286.07080078125$ memory_percent‚90.83101749420166) proc.memory.availableMB‚ 1502.24609375 disk.nvme0c0n1.in‚ 37.484375 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚19.641780853271484 cpu‚16.90419703231548‡S¼ !‚(:“ ²èÔ‹ÿÿà gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.08113098144531* gpu.0.memoryAllocatedBytes‚ 58481246208 gpu.0.temp‚41.0 gpu.0.powerWatts‚247.172' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.434400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ec+±’"‚(:ˆ ÁèÔ ÎäÞ gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.08601379394531* gpu.0.memoryAllocatedBytes‚ 58485440512 gpu.0.temp‚41.0 gpu.0.powerWatts‚241.258' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.2516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ZÁc#‚(:Ù ÁèÔâ‚ß# memory_percent‚90.8322811126709 disk.sda1.out‚0 disk.drbd1260.out‚ 0.00390625 disk.sda3.in‚0& disk.nvme0c0n1.out‚704.81689453125 disk.sda.in‚0# proc.memory.rssMB‚ 3218.22265625 cpu‚17.96311431414385 proc.cpu.threads‚17 network.sent‚51052 disk.sda2.out‚0! disk.dm-0.out‚704.81689453125$ disk.nvme0n1.out‚704.81689453125( proc.memory.availableMB‚ 1502.0390625 network.recv‚49589 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 150.41796875" disk.nvme0c0n1.in‚ 150.41796875 disk.sda.out‚ 1.48828125) proc.memory.percent‚19.64247226715088) disk./.usagePercent‚77.93518577381288% disk./.usageGB‚1161.2905883789062 disk.sda1.in‚0 disk.dm-1.out‚ 1.48828125 disk.dm-0.in‚ 150.41796875 disk.sda3.out‚ 1.48828125&˜¹a$‚(:× ÐèÔ—ôØÞ disk.sda3.in‚0 disk.sda3.out‚ 1.8671875 network.recv‚79911) disk./.usagePercent‚77.94360534959898 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚955.85498046875$ memory_percent‚90.84303379058838) proc.memory.availableMB‚ 1500.27734375& disk.nvme0c0n1.out‚955.85498046875 disk.sda.out‚ 1.8671875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 150.41796875# proc.memory.rssMB‚ 3218.23828125% disk./.usageGB‚1161.4160461425781 disk.sda.in‚0 disk.nvme0n1.in‚ 150.41796875) proc.memory.percent‚19.64256763458252 cpu‚18.66177455237677 proc.cpu.threads‚17 network.sent‚70749" disk.nvme0c0n1.in‚ 150.41796875 disk.dm-1.out‚ 1.8671875! disk.dm-0.out‚955.85498046875²æÜ%‚(:“ ÐèÔ¿‘Þß gpu.0.gpu‚59.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚68.09089660644531* gpu.0.memoryAllocatedBytes‚ 58489634816 gpu.0.temp‚39.0 gpu.0.powerWatts‚135.651' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.130200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æèªe&‚(:Û ßèÔŽª¼à( proc.memory.availableMB‚ 1499.4453125 disk.dm-0.in‚ 150.41796875% disk.nvme0n1.out‚1398.57373046875 disk.sda3.in‚0' disk.nvme0c0n1.out‚1398.57373046875! proc.memory.rssMB‚ 3218.265625 proc.cpu.threads‚17 disk.sda2.out‚0 network.sent‚76338 network.recv‚84151% disk./.usageGB‚1161.9400177001953 disk.sda.out‚ 2.61328125 disk.sda1.out‚0 disk.dm-1.out‚ 2.61328125" disk.nvme0c0n1.in‚ 150.41796875) disk./.usagePercent‚77.97876952047208 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 2.61328125) proc.memory.percent‚19.64273452758789 cpu‚19.338510640838646 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0078125 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚1398.57373046875 disk.nvme0n1.in‚ 150.41796875$ memory_percent‚90.84811210632324iºƒ#’'‚(:ˆ ßèÔêž á gpu.0.gpu‚95.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.09333801269531* gpu.0.memoryAllocatedBytes‚ 58491731968 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÝíœY(‚(:Ï îèÔ»ïùÞ disk.dm-0.out‚ 1744.28515625" disk.nvme0n1.out‚ 1744.28515625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 150.41796875 disk.sda.out‚ 3.2890625 disk.sda1.out‚0 disk.nvme0n1.in‚ 150.41796875 disk.sda3.out‚ 3.2890625 disk.sda.in‚0 disk.dm-1.out‚ 3.2890625 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.84734916687012% disk./.usageGB‚1161.9389991760254 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.01171875 cpu‚19.894781798436817 proc.cpu.threads‚17 network.sent‚82416) disk./.usagePercent‚77.97870116646044$ disk.nvme0c0n1.out‚ 1744.28515625( proc.memory.availableMB‚ 1499.5703125 proc.memory.rssMB‚ 3218.28125) proc.memory.percent‚19.64282989501953 network.recv‚88313 disk.dm-0.in‚ 150.41796875“Û`)‚(:“ îèÔÔ§ñß gpu.0.gpu‚88.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.09577941894531* gpu.0.memoryAllocatedBytes‚ 58493829120 gpu.0.temp‚43.0 gpu.0.powerWatts‚243.379' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.675799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÓòH]*‚(:Ó ýèÔÊĹÞ network.sent‚89217 disk.drbd1260.in‚0 disk.dm-0.out‚ 2124.87109375" disk.nvme0n1.out‚ 2124.87109375 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 2124.87109375$ memory_percent‚90.85087776184082 proc.cpu.threads‚17 disk.sda2.in‚0 network.recv‚93668) disk./.usagePercent‚77.97870116646044% disk./.usageGB‚1161.9389991760254 disk.sda.in‚0 disk.sda.out‚ 3.6328125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.01171875 disk.sda1.in‚0 disk.dm-1.out‚ 3.6328125 disk.nvme0n1.in‚ 279.08984375 disk.sda3.out‚ 3.6328125( proc.memory.availableMB‚ 1498.9921875# proc.memory.rssMB‚ 3218.30078125* proc.memory.percent‚19.642949104309082 cpu‚20.351370043031526 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 279.08984375" disk.nvme0c0n1.in‚ 279.08984375“@\ó‘+‚(:‡ ýèÔŸâ½à gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.09822082519531* gpu.0.memoryAllocatedBytes‚ 58495926272 gpu.0.temp‚44.0 gpu.0.powerWatts‚248.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.693 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åbu],‚(:Ó ŒžèÔû‹«ß disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 3.95703125 proc.memory.rssMB‚ 3218.3125 network.recv‚97937% disk./.usageGB‚1162.4155311584473 disk.dm-0.out‚2183.494140625 disk.sda3.out‚ 3.95703125" disk.nvme0c0n1.in‚ 279.08984375 disk.drbd1260.out‚0.015625 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚94722 disk.dm-0.in‚ 279.08984375 disk.nvme0n1.in‚ 279.08984375 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚90.85047245025635) proc.memory.availableMB‚ 1499.05859375* proc.memory.percent‚19.643020629882812) disk./.usagePercent‚78.01068162763774# disk.nvme0n1.out‚2183.494140625% disk.nvme0c0n1.out‚2183.494140625 disk.drbd1260.in‚0 cpu‚20.74266522812286 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.out‚ 3.95703125ϜՒ-‚(:ˆ ŒžèÔå™Ãà gpu.0.gpu‚91.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.10066223144531* gpu.0.memoryAllocatedBytes‚ 58498023424 gpu.0.temp‚43.0 gpu.0.powerWatts‚231.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.2212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0; U’.‚(:ˆ ›žèÔÆß®Þ gpu.0.gpu‚98.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚68.10066223144531* gpu.0.memoryAllocatedBytes‚ 58498023424 gpu.0.temp‚48.0 gpu.0.powerWatts‚336.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚058t'_/‚(:Õ ›žèÔšêÎß disk.sda3.out‚ 4.3984375 disk.drbd1260.out‚0.015625( proc.memory.availableMB‚ 1498.8984375 network.recv‚102185 disk.sda2.in‚0 disk.dm-0.in‚ 279.08984375 disk.nvme0n1.in‚ 279.08984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 4.3984375$ memory_percent‚90.85144996643066 disk.sda2.out‚0% disk.nvme0c0n1.out‚2554.154296875 disk.sda.in‚0 disk.sda.out‚ 4.3984375 disk.dm-1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3218.32421875* proc.memory.percent‚19.643092155456543 disk.dm-0.out‚2554.154296875" disk.nvme0c0n1.in‚ 279.08984375 cpu‚21.074706143454392 proc.cpu.threads‚17 network.sent‚100317) disk./.usagePercent‚78.04594743350194$ disk./.usageGB‚1162.941017150879# disk.nvme0n1.out‚2554.154296875 disk.sda3.in‚0ÄyHk0‚(:á ªžèÔËÝïÝ network.recv‚106301 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 279.08984375% disk.nvme0n1.out‚3165.89306640625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 279.08984375% disk./.usageGB‚1162.9389991760254 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 4.77734375 disk.drbd1260.out‚ 0.01953125 disk.sda.in‚0) proc.memory.availableMB‚ 1498.80859375# proc.memory.rssMB‚ 3218.32421875 network.sent‚105823 disk.sda2.out‚0" disk.dm-0.out‚3165.89306640625 disk.sda3.out‚ 4.77734375' disk.nvme0c0n1.out‚3165.89306640625 disk.sda.out‚ 4.77734375) disk./.usagePercent‚78.04581200551632 disk.dm-0.in‚ 279.08984375# memory_percent‚90.8519983291626* proc.memory.percent‚19.643092155456543 cpu‚21.365160257528345 proc.cpu.threads‚17é @œ1‚(:’ ªžèÔý¨ß gpu.0.gpu‚13.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚68.10310363769531* gpu.0.memoryAllocatedBytes‚ 58500120576 gpu.0.temp‚39.0 gpu.0.powerWatts‚135.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʉèO32b/ lr-AdamW/pg1"trainer/global_step2:JPac_¦33b/ lr-AdamW/pg2"trainer/global_step2:JPµ:v/4+ _wandbruntime‚161  _runtime‚161ðÓ÷PÁ5¼ trainer/global_step‚49 % _timestamp‚1.7884813343497307e+09 _runtime‚ 161.895060393 _step‚0 & lr-AdamW/pg1‚8.222803967627718e-05 & lr-AdamW/pg2‚8.222803967627718e-05ó}f‘Ã6¾ & lr-AdamW/pg1‚8.222803967627718e-05 & lr-AdamW/pg2‚8.222803967627718e-05 trainer/global_step‚49 % _timestamp‚1.7884813343497307e+09 _runtime‚ 161.895060393 _step‚0ÚÞüÿm7bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP]écr8bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÙ€pÔk9bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPbä(s:bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP`šh.i;be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPaZ˜jb_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPµ5ˆgAbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPJfŠÆeBba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPB'ÔÂsCbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP™!ØjDbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPuzŠümEbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPQcáÙkFbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP ìqGbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPe5½fHbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPÔøågIbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP:7XkJbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPìÖ=pKbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPwÆtgLbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP˜û uqMbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPè.ËkNbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP@˜ ·vObr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPµ†?µ^PbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP„ðÎiQbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP£kRbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP_~ölSbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÄ£vZjTbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPéд–gUbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP«nvbVb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP´#äünWbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPè§ómXbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPü#ÝnYbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP¸1¯iZbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP´Ñ½__[b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPè¡¥úi\be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP7UÓ¬i]be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPÇCs¾x^bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPõJÄ"t_bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPþ¡[g`bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP!Tñ³fabb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP–Ðüõfbbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPrGˆ¼ncbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP014|rdbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPÆaû&nebj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPÓs:¢qfbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPq5……egba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP-C,9chb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP¿Ç`€b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPb9eXbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP6rtWk‚bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP2M0kƒbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP ¯n„bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPµ˜åp…bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP¼ÓÃão†bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP[”G7s‡bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPe'^'gˆbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP åƒg‰bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPŠŒ‹wŠbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP›ç6Xh‹bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPc®hŒbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPöOÏgbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPiû9…bŽb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPÝQ`Ùmbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPrÃçfba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP¿‡‘r‘bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPЩ2Å^’bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP–sgl“bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP5\aq”bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP}°ýo•bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPš³„m–bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¨Ø'o—bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPHB÷[q˜bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP­ ´)a™b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP«]åÝišbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPlèEp›bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPOXÎAuœbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP PYbb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP̦æ»mžbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP0âB`Ÿb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP^#Xâj be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPë¸/¿g¡bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPà ×_¢bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPŸJ²ªt£bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPO”­®i¤bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPeƒb’l¥bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP±ò&m¦bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP¾Ü÷i§bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPÁ2Üþj¨be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPJWDüt©bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPã+\oªbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPÁãîj«be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP‹wGwi¬bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPpðÄ#i­bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP“\t3o®bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPæ̳q¯bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP ZC*p°bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPÅÅLo±bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPŽäÓÿ0²+ _wandbruntime‚161  _runtime‚161ž.Â'³¼O _step‚1 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30386 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5305 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1209043 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 120460288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚286978 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323341 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2169 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3583163203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57946406912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15815027110400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 148055780864 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚286975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 147850683704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 194944442880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15328740811248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5305 trainer/global_step‚49 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15966651059200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57946406912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30635 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 151623948800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15815001682432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15963057463296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 151417514720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57946406912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 151623948800 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 148055780864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456803328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4548722688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1023111 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15966651059200 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15963057463296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15815027110400 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1023105 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3583042742784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2169 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4548722688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018240000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚713645 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚91362304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456803328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1650948608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2256944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30635 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1233833 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2232148 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30386 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚803 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3388127397888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3388098299904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1023105 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1209043 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚426667 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚425867’ zúÆ'´ÀO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30386 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5305 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1209043 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 120460288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚286978 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323341 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2169 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3583163203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57946406912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15815027110400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 148055780864 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚286975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 147850683704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 194944442880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15328740811248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5305 trainer/global_step‚49 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15966651059200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57946406912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30635 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 151623948800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15815001682432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15963057463296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 151417514720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57946406912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 151623948800 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 148055780864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456803328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4548722688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1023111 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15966651059200 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15963057463296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15815027110400 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1023105 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3583042742784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2169 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4548722688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018240000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚713645 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚91362304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456803328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1650948608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2256944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30635 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1233833 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2232148 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30386 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚803 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3388127397888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3388098299904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1023105 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1209043 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚426667 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚425867 _step‚1 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPçËšiÂbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP'Ä jÃbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPA«÷.mÄbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP¸×ܼmÅbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPi1²^ÆbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPñ'G›eÇb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPà›èTrÈbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPáÞ‘_ÉbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP$Q/KfÊba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPCÚkåhËbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP·ºýÂmÌbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP$DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP>U^oïbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPÙÇ“nðbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPu‡·Ëpñbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPòúqyfòba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP)¨9DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPŸ'ŠjŠbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPANe:f‹ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP 5zfŒba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP¨¤‚]db_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP9ö«?b§b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP¸mw×b¨b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPXyŠq©bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP•1ûGsªbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP} d«b_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2301512 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚727639 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚726836 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 165.735165434 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2301512 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2276690 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚435177 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 198806174720 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1258151 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5305 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚434377 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57946406912 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16280643066368 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1043355 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154618548736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4548722688 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15478968865760 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2301512 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚727639 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚726836 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 165.735165434 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2301512 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2276690 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚435177 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 198806174720 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1258151 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5305 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚434377 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57946406912 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16280643066368 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1043355 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154618548736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4548722688 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15478968865760 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E loss/SRqqt3/assignment_loss"trainer/global_step2:JPnA#B¿b= loss/SRqqt3/detection_loss"trainer/global_step2:JPÕÉqAÀb< loss/FRt3/assignment_loss"trainer/global_step2:JP»}ÂGCÁb> loss/SRqqt2/assignment_loss"trainer/global_step2:JPAb»MAÂb< loss/FBt1/assignment_loss"trainer/global_step2:JPwzAÃb< loss/FBt3/assignment_loss"trainer/global_step2:JPÀ¡VÏ@Äb; loss/FRt1/detection_loss"trainer/global_step2:JPÄ@mBÅb= loss/SRqqt2/detection_loss"trainer/global_step2:JP÷]1@Æb; loss/FBt2/detection_loss"trainer/global_step2:JPê¹äCÇb> loss/SRqqt1/assignment_loss"trainer/global_step2:JP;걂CÈb> loss/SRqqt4/assignment_loss"trainer/global_step2:JPg|\AÉb< loss/FBt4/assignment_loss"trainer/global_step2:JP ÝEÚ@Êb; loss/FRt3/detection_loss"trainer/global_step2:JP*¶Ÿ@Ëb; loss/FBt3/detection_loss"trainer/global_step2:JPT 0AÌb< loss/FRt2/assignment_loss"trainer/global_step2:JPKàt=@Íb; loss/FRt2/detection_loss"trainer/global_step2:JP )Ì“@Îb; loss/FBt1/detection_loss"trainer/global_step2:JP±©²œ0Ï+ _wandbruntime‚165  _runtime‚165±;/|MÐÇ /loss/FBt4/detection_loss‚1.8891079689742687 _runtime‚ 165.740751224 1loss/SRqqt3/assignment_loss‚5.491713636892813 1loss/SRqqt3/detection_loss‚1.8626802868951389 .loss/FRt3/assignment_loss‚7.34530975988933 1loss/SRqqt2/assignment_loss‚5.451324517910297 0loss/FBt1/assignment_loss‚1.3125412662823994 0loss/FBt3/assignment_loss‚0.9983369411998674 /loss/FRt1/detection_loss‚3.9619652446020734 0loss/SRqqt2/detection_loss‚4.487390746290867 _step‚3 .loss/FBt2/detection_loss‚9.775544609342303 trainer/global_step‚49 1loss/SRqqt1/assignment_loss‚5.378949668429313 1loss/SRqqt4/assignment_loss‚5.243148463112967 0loss/FBt4/assignment_loss‚0.4644342439849515 .loss/FRt3/detection_loss‚3.897074604673045 .loss/FBt3/detection_loss‚9.335614404396006 % _timestamp‚1.7884813381984496e+09 0loss/FRt2/assignment_loss‚7.4184402873118716 -loss/FRt2/detection_loss‚3.57215586444363 /loss/FBt1/detection_loss‚13.970540756980578 0loss/FBt2/assignment_loss‚1.1215739233153208 &loss/total_loss‚115.98413673432577 epoch‚0 /loss/FRt1/assignment_loss‚7.379381759600206 /loss/FRt4/assignment_loss‚7.205141719589886 /loss/FRt4/detection_loss‚2.6847351397523003 1loss/SRqqt1/detection_loss‚3.2027107939260815 1loss/SRqqt4/detection_loss‚2.5343201265302526¿ ØÙQÑË _step‚3 .loss/FBt2/detection_loss‚9.775544609342303 trainer/global_step‚49 1loss/SRqqt1/assignment_loss‚5.378949668429313 1loss/SRqqt4/assignment_loss‚5.243148463112967 0loss/FBt4/assignment_loss‚0.4644342439849515 .loss/FRt3/detection_loss‚3.897074604673045 .loss/FBt3/detection_loss‚9.335614404396006 % _timestamp‚1.7884813381984496e+09 0loss/FRt2/assignment_loss‚7.4184402873118716 -loss/FRt2/detection_loss‚3.57215586444363 /loss/FBt1/detection_loss‚13.970540756980578 0loss/FBt2/assignment_loss‚1.1215739233153208 &loss/total_loss‚115.98413673432577 epoch‚0 /loss/FRt1/assignment_loss‚7.379381759600206 /loss/FRt4/assignment_loss‚7.205141719589886 /loss/FRt4/detection_loss‚2.6847351397523003 1loss/SRqqt1/detection_loss‚3.2027107939260815 1loss/SRqqt4/detection_loss‚2.5343201265302526 /loss/FBt4/detection_loss‚1.8891079689742687 _runtime‚ 165.740751224 1loss/SRqqt3/assignment_loss‚5.491713636892813 1loss/SRqqt3/detection_loss‚1.8626802868951389 .loss/FRt3/assignment_loss‚7.34530975988933 1loss/SRqqt2/assignment_loss‚5.451324517910297 0loss/FBt1/assignment_loss‚1.3125412662823994 0loss/FBt3/assignment_loss‚0.9983369411998674 /loss/FRt1/detection_loss‚3.9619652446020734 0loss/SRqqt2/detection_loss‚4.487390746290867EÎ-ådÒ‚(:Ù ÈžèÔ﬈ß# proc.memory.rssMB‚ 3218.34765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 5.7265625 disk.drbd1260.out‚ 0.0234375 disk.dm-0.in‚ 415.3203125 disk.sda3.in‚0 disk.sda3.out‚ 5.7265625 proc.cpu.threads‚17 network.sent‚146826$ disk./.usageGB‚1162.939121246338! disk.nvme0c0n1.in‚ 415.3203125' disk.nvme0c0n1.out‚3954.64208984375 disk.sda2.in‚0 disk.nvme0n1.in‚ 415.3203125* proc.memory.percent‚19.643235206604004 cpu‚21.83897662333251 network.recv‚119513 disk.sda.in‚0$ memory_percent‚90.85605144500732) proc.memory.availableMB‚ 1498.14453125) disk./.usagePercent‚78.04582019775742 disk.sda.out‚ 5.7265625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.dm-0.out‚3954.64208984375% disk.nvme0n1.out‚3954.64208984375//9a“Ó‚(:ˆ ÈžèԽؒá gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10310363769531* gpu.0.memoryAllocatedBytes‚ 58500120576 gpu.0.temp‚44.0 gpu.0.powerWatts‚244.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n<9bÔ‚(:× מèԌ÷à% disk./.usageGB‚1162.9390869140625! disk.nvme0c0n1.in‚ 415.3203125 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3218.34765625* proc.memory.percent‚19.643235206604004 cpu‚22.041866593926123 network.sent‚185447 network.recv‚124500 disk.sda3.out‚6.171875 disk.sda.out‚6.171875 proc.cpu.threads‚17) disk./.usagePercent‚78.04581789368962 disk.drbd1260.out‚ 0.0234375 disk.sda2.out‚0% disk.nvme0n1.out‚4167.98193359375 disk.dm-1.in‚0 disk.dm-1.out‚6.171875$ memory_percent‚90.85803031921387( proc.memory.availableMB‚ 1497.8203125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 415.3203125" disk.dm-0.out‚4167.98193359375 disk.nvme0n1.in‚ 415.3203125 disk.sda3.in‚0' disk.nvme0c0n1.out‚4167.98193359375 disk.sda.in‚0¥8¢ªzÕ‚(:ˆ מèÔîÕˆá gpu.0.gpu‚54.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.10310363769531* gpu.0.memoryAllocatedBytes‚ 58500120576 gpu.0.temp‚39.0 gpu.0.powerWatts‚115.147' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.un s[ïcorrectedMemoryErrors‚05ÿc hÖ‚(:Ý æžèÔªÀîÞ% disk.nvme0n1.out‚4465.83544921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.02734375$ memory_percent‚90.85683822631836# proc.memory.rssMB‚ 3218.34765625 network.recv‚128298" disk.dm-0.out‚4465.83544921875 disk.sda.out‚ 6.64453125 disk.sda1.out‚0 disk.dm-1.out‚ 6.64453125 disk.sda2.out‚0 network.sent‚190853 disk.nvme0n1.in‚ 415.3203125 disk.sda3.in‚0 disk.sda3.out‚ 6.64453125! disk.nvme0c0n1.in‚ 415.3203125' disk.nvme0c0n1.out‚4465.83544921875 disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚78.04581661365194% disk./.usageGB‚1162.9390678405762 disk.dm-1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 1498.015625* proc.memory.percent‚19.643235206604004 cpu‚22.217763109767958 proc.cpu.threads‚17 disk.dm-0.in‚ 415.3203125`&þ“ׂ(:ˆ æžèÔ–ÉÈß gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚47.0 gpu.0.powerWatts‚322.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.5488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°û^[Ø‚(:Ð õžèÔ±ôˆß" disk.dm-0.out‚4884.49560546875% disk.nvme0n1.out‚4884.49560546875 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚7.0625* proc.memory.percent‚19.643235206604004) disk./.usagePercent‚78.04424165530129 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.02734375 disk.sda2.out‚0' disk.nvme0c0n1.out‚4884.49560546875 cpu‚22.37832014295478 proc.cpu.threads‚17 network.sent‚197591 disk.sda3.out‚7.0625! disk.nvme0c0n1.in‚ 415.3203125 disk.sda1.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 1497.9765625 network.recv‚133532 disk.dm-0.in‚ 415.3203125 disk.nvme0n1.in‚ 415.3203125 disk.sda.out‚7.0625 disk.sda1.in‚0$ memory_percent‚90.85707664489746# proc.memory.rssMB‚ 3218.34765625$ disk./.usageGB‚1162.915599822998 disk.sda2.in‚0C'ë“Ù‚(:ˆ õžèÔõ´®ß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚44.0 gpu.0.powerWatts‚218.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.6026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðrv%hÚ‚(:Ý „ŸèÔɈàÞ% disk.nvme0n1.out‚5123.08154296875 disk.sda.out‚ 7.43359375 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚138202' disk.nvme0c0n1.out‚5123.08154296875 disk.dm-1.out‚ 7.43359375# memory_percent‚90.8531665802002# proc.memory.rssMB‚ 3218.34765625* proc.memory.percent‚19.643235206604004 network.sent‚205765 disk.drbd1260.out‚0.03125 disk.dm-0.in‚ 586.15234375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 586.15234375 disk.sda.in‚0( proc.memory.availableMB‚ 1498.6171875 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚5123.08154296875 disk.sda3.out‚ 7.43359375 disk.sda1.out‚0 cpu‚22.525262647846844 proc.cpu.threads‚17) disk./.usagePercent‚78.04581661365194% disk./.usageGB‚1162.9390678405762 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 586.15234375×ÞðÒ’Û‚(:‡ „ŸèÔÉÁÃß gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚44.0 gpu.0.powerWatts‚246.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.271 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ºY!fÜ‚(:Û “ŸèÔ·¯þÞ) disk./.usagePercent‚78.04581661365194 disk.sda.out‚ 8.0703125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 8.0703125 disk.sda2.in‚0" disk.dm-0.out‚5565.81591796875 network.sent‚209611% disk./.usageGB‚1162.9390678405762" disk.nvme0c0n1.in‚ 586.15234375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 586.15234375% disk.nvme0n1.out‚5565.81591796875 network.recv‚142612' disk.nvme0c0n1.out‚5565.81591796875 disk.dm-0.in‚ 586.15234375 disk.sda3.in‚0) proc.memory.availableMB‚ 1498.85546875" proc.memory.rssMB‚ 3218.3515625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1260.out‚0.03125 disk.sda3.out‚ 8.0703125$ memory_percent‚90.85171222686768* proc.memory.percent‚19.643259048461914 cpu‚22.657272243713557=z¦žÝ‚(:“ “ŸèÔ¢¨³ß gpu.0.gpu‚92.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚43.0 gpu.0.powerWatts‚237.776' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.555200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹°$gÞ‚(:Ü ¢ŸèÔÁ¦Šß" disk.nvme0c0n1.in‚ 586.15234375 disk.sda.out‚ 8.56640625 disk.sda1.in‚0 disk.dm-1.out‚ 8.56640625 disk.sda2.out‚0 disk.dm-0.in‚ 586.15234375 disk.sda3.out‚ 8.56640625) disk./.usagePercent‚78.04581661365194% disk./.usageGB‚1162.9390678405762 disk.sda1.out‚0 disk.drbd1260.out‚ 0.0390625$ memory_percent‚90.85183143615723( proc.memory.availableMB‚ 1498.8359375" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914 network.recv‚146518& disk.nvme0c0n1.out‚5791.3720703125 disk.nvme0n1.in‚ 586.15234375$ disk.nvme0n1.out‚5791.3720703125 disk.sda3.in‚0 cpu‚22.778556442570718 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.dm-0.out‚5791.3720703125 network.sent‚214949³’ß‚(:‡ ¢ŸèÔ”»Øà gpu.0.gpu‚44.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚41.0 gpu.0.powerWatts‚193.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖdÀ5dà‚(:Ù ±ŸèÔ©ýùß& disk.nvme0c0n1.out‚6099.7626953125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 8.8984375 disk.drbd1260.out‚ 0.0390625 proc.cpu.threads‚17) disk./.usagePercent‚78.04424165530129 disk.sda3.in‚0 disk.sda.out‚ 8.8984375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 586.15234375! disk.dm-0.out‚6099.7626953125 disk.nvme0n1.in‚ 586.15234375$ memory_percent‚90.85185527801514) proc.memory.availableMB‚ 1498.83203125 network.sent‚221737 network.recv‚151750 disk.sda3.out‚ 8.8984375 disk.sda2.in‚0$ disk.nvme0n1.out‚6099.7626953125" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914 cpu‚22.888947578820964$ disk./.usageGB‚1162.915599822998" disk.nvme0c0n1.in‚ 586.15234375„*äøžá‚(:“ ±ŸèÔù—®â gpu.0.gpu‚47.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚43.0 gpu.0.powerWatts‚199.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰¶X`â‚(:Õ ÀŸèÔ«Ôšß network.sent‚227964) disk./.usagePercent‚78.01667962815745% disk./.usageGB‚1162.5049057006836 disk.dm-1.out‚9.484375# disk.nvme0n1.out‚6542.693359375% disk.nvme0c0n1.out‚6542.693359375" proc.memory.rssMB‚ 3218.3515625 disk.dm-0.in‚ 714.96484375" disk.nvme0c0n1.in‚ 714.96484375 disk.sda.out‚9.484375 cpu‚22.990720317415235 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚9.484375) proc.memory.availableMB‚ 1498.32421875* proc.memory.percent‚19.643259048461914 network.recv‚156535 disk.sda1.in‚0 disk.drbd1260.out‚ 0.04296875 disk.sda2.in‚0 disk.nvme0n1.in‚ 714.96484375 disk.sda.in‚0$ memory_percent‚90.85495471954346 disk.dm-0.out‚6542.693359375ã ÿ“ã‚(:ˆ ÀŸèÔíØ¢à gpu.0.gpu‚60.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚44.0 gpu.0.powerWatts‚253.559' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äøq0ä+ _wandbruntime‚314  _runtime‚314MË’ÚÂå¼ & lr-AdamW/pg1‚0.0001661342026112539 & lr-AdamW/pg2‚0.0001661342026112539 trainer/global_step‚99 % _timestamp‚1.7884814865233798e+09 _runtime‚ 314.065365885 _step‚4 ÆæÀ % _timestamp‚1.7884814865233798e+09 _runtime‚ 314.065365885 _step‚4 & lr-AdamW/pg1‚0.0001661342026112539 & lr-AdamW/pg2‚0.0001661342026112539 trainer/global_step‚99¸ùsÊ0ç+ _wandbruntime‚314  _runtime‚314±¥LÅÃ'è½O RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2426995 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2451785 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 733117952 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2035611 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31544577932800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4462600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 314.06949043 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2035611 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚851079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 297403486412 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4550819840 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2170 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31842365445120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2170 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 301381108224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 387521647104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456804352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2035605 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 387426090496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4550819840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30260873156528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 300970317428 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5306 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 301381108224 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6852818188800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95556608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2426995 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚561175 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4487396 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6465421196288 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4550819840 trainer/global_step‚99 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6852942843392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5306 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30260847728560 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31845959041024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1644 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚832 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚850247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30558251214972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31544552504832 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚561178 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5306 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2170 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚835 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790k©ÖÇ'éÁO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4462600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 314.06949043 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2035611 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚851079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 297403486412 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4550819840 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2170 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31842365445120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2170 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 301381108224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 387521647104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456804352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2035605 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 387426090496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4550819840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30260873156528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 300970317428 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5306 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 301381108224 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6852818188800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95556608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2426995 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚561175 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4487396 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6465421196288 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4550819840 trainer/global_step‚99 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6852942843392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5306 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30260847728560 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31845959041024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1644 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚832 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚850247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30558251214972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31544552504832 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚561178 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5306 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2170 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚835 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2426995 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2451785 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 733117952 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2035611 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31544577932800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968;¼ ¹hê‚(:Ý ÏŸèÔ“£àß disk.drbd1260.out‚ 0.04296875 disk.sda2.out‚0 disk.dm-0.out‚6913.470703125( proc.memory.availableMB‚ 1498.3984375* proc.memory.percent‚19.643259048461914 cpu‚23.081313084283654 proc.cpu.threads‚17 disk.drbd1260.in‚0 network.sent‚233446% disk./.usageGB‚1161.8774452209473" disk.nvme0c0n1.in‚ 714.96484375 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3218.3515625 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 10.03515625 disk.sda3.in‚0 disk.sda3.out‚ 10.03515625 network.recv‚160559) disk./.usagePercent‚77.97457022888794 disk.sda.out‚ 10.03515625 disk.sda2.in‚0 disk.dm-0.in‚ 714.96484375 disk.nvme0n1.in‚ 714.96484375# disk.nvme0n1.out‚6913.470703125$ memory_percent‚90.85450172424316% disk.nvme0c0n1.out‚6913.470703125!Î „“ë‚(:ˆ ÏŸèÔ¨Ïüß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚46.0 gpu.0.powerWatts‚251.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ôh0ì+ _wandbruntime‚316  _runtime‚316RJÓ÷ÆíÈM QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2476103 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1678210560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4550819840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 300807526912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 391393916928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31859143521280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 304375708160 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2055855 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStat Ö#‚!sMonitor.on_train_batch_end/segment.small_pool.allocated‚2170 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚823 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4507142 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4507142 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1306 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2170 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4531964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 303960830250 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31859168949248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31859168949248 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1426249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57948504064 _runtime‚ 316.256044745 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57948504064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57948504064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5306 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚566662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 733117952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2476103 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 304375708160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32159951048192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚566659 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 300393999026 trainer/global_step‚99 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31859143521280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 122544128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30866504983978 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2055861 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30862912724786 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5306 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6918238445056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2055855 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2055861 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚858764 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6526967072256 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 391300470784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2451287 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1425423 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚93446144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 _step‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚826 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5306 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2170 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 300807526912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5306 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4531964 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32163544657408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4550819840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30562544153728 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30562518725760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4550819840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6918360989184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248Üñ=Ò&îÌM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1426249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57948504064 _runtime‚ 316.256044745 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57948504064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57948504064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5306 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚566662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 733117952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2476103 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 304375708160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32159951048192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚566659 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 300393999026 trainer/global_step‚99 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31859143521280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 122544128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30866504983978 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2055861 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30862912724786 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5306 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6918238445056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2055855 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2055861 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚858764 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6526967072256 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 391300470784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2451287 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1425423 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚93446144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 _step‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚826 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5306 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2170 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 300807526912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5306 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4531964 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32163544657408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4550819840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30562544153728 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30562518725760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4550819840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6918360989184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2476103 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1678210560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4550819840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 300807526912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 391393916928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31859143521280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 304375708160 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2055855 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2170 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚823 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4507142 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4507142 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1306 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2170 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4531964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 303960830250 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31859168949248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31859168949248ÏÉã­0ï+ _wandbruntime‚316  _runtime‚316Ð H²NðÈ 0loss/FRt3/assignment_loss‚7.1678057988484705 1loss/SRqqt4/detection_loss‚1.1352250752518263 trainer/global_step‚99 2loss/SRqqt2/assignment_loss‚3.6722999878649443 /loss/FBt1/detection_loss‚12.085901315565463 .loss/FBt4/detection_loss‚4.920534729374734 % _timestamp‚1.7884814887165504e+09 /loss/FRt4/assignment_loss‚7.208128144190861 0loss/FBt3/assignment_loss‚0.5658556463132052 0loss/SRqqt2/detection_loss‚6.371251691062495 0loss/SRqqt3/detection_loss‚4.999045406214216 _step‚7 /loss/FRt1/assignment_loss‚7.430779311236213 1loss/SRqqt4/assignment_loss‚3.425606027530257 .loss/FRt3/detection_loss‚4.359107553958893 0loss/SRqqt1/detection_loss‚3.205168731212616 .loss/FBt3/detection_loss‚6.182724391872233 1loss/FBt1/assignment_loss‚0.39856420061492087 -loss/FBt2/detection_loss‚4.04685860586493 epoch‚0 /loss/FRt2/detection_loss‚3.2485131079902745 /loss/FRt2/assignment_loss‚7.389400907082133 2loss/SRqqt1/assignment_loss‚3.2601497626304625 1loss/SRqqt3/assignment_loss‚3.619342915939562 0loss/FBt2/assignment_loss‚0.1138216916698022 /loss/FRt1/detection_loss‚3.9343417644500733 &loss/total_loss‚101.54265581885973 _runtime‚ 316.257836452 0loss/FBt4/assignment_loss‚0.7600818780734107 .loss/FRt4/detection_loss‚2.042147174047736Ξ^ïRñÌ _runtime‚ 316.257836452 0loss/FBt4/assignment_loss‚0.7600818780734107 .loss/FRt4/detection_loss‚2.042147174047736 0loss/FRt3/assignment_loss‚7.1678057988484705 1loss/SRqqt4/detection_loss‚1.1352250752518263 trainer/global_step‚99 2loss/SRqqt2/assignment_loss‚3.6722999878649443 /loss/FBt1/detection_loss‚12.085901315565463 .loss/FBt4/detection_loss‚4.920534729374734 % _timestamp‚1.7884814887165504e+09 /loss/FRt4/assignment_loss‚7.208128144190861 0loss/FBt3/assignment_loss‚0.5658556463132052 0loss/SRqqt2/detection_loss‚6.371251691062495 0loss/SRqqt3/detection_loss‚4.999045406214216 _step‚7 /loss/FRt1/assignment_loss‚7.430779311236213 1loss/SRqqt4/assignment_loss‚3.425606027530257 .loss/FRt3/detection_loss‚4.359107553958893 0loss/SRqqt1/detection_loss‚3.205168731212616 .loss/FBt3/detection_loss‚6.182724391872233 1loss/FBt1/assignment_loss‚0.39856420061492087 -loss/FBt2/detection_loss‚4.04685860586493 epoch‚0 /loss/FRt2/detection_loss‚3.2485131079902745 /loss/FRt2/assignment_loss‚7.389400907082133 2loss/SRqqt1/assignment_loss‚3.2601497626304625 1loss/SRqqt3/assignment_loss‚3.619342915939562 0loss/FBt2/assignment_loss‚0.1138216916698022 /loss/FRt1/detection_loss‚3.9343417644500733 &loss/total_loss‚101.54265581885973½¥Ï]kò‚(:à ÞŸèÔŸÛêÝ* proc.memory.percent‚19.643259048461914 cpu‚23.164257484390344) disk./.usagePercent‚77.97454488414205" disk.nvme0c0n1.in‚ 714.96484375 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚165609 disk.sda.out‚ 10.69921875 disk.dm-1.out‚ 10.69921875 disk.drbd1260.out‚0.046875 disk.sda2.in‚0 disk.dm-0.in‚ 714.96484375" disk.dm-0.out‚7371.84716796875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 714.96484375% disk.nvme0n1.out‚7371.84716796875$ memory_percent‚90.85488319396973( proc.memory.availableMB‚ 1498.3359375 network.sent‚271208$ disk./.usageGB‚1161.877067565918' disk.nvme0c0n1.out‚7371.84716796875 disk.drbd1260.in‚0 disk.sda3.out‚ 10.69921875" proc.memory.rssMB‚ 3218.3515625.d1“ó‚(:ˆ ÞŸèÔŸëùÞ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚46.0 gpu.0.powerWatts‚232.459' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`vôfô‚(:Û íŸèÔ«®ßß' disk.nvme0c0n1.out‚7659.42529296875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 11.3046875$ memory_percent‚90.85359573364258' proc.memory.availableMB‚ 1498.546875 cpu‚23.24143916127077 proc.cpu.threads‚17 disk.nvme0n1.in‚ 714.96484375% disk.nvme0n1.out‚7659.42529296875 disk.sda3.out‚ 11.3046875 network.sent‚310898$ disk./.usageGB‚1161.877109527588" disk.nvme0c0n1.in‚ 714.96484375 disk.sda.out‚ 11.3046875 disk.sda1.in‚0) disk./.usagePercent‚77.97454770022493" disk.dm-0.out‚7659.42529296875 disk.sda3.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914 network.recv‚171760 disk.drbd1260.in‚0 disk.drbd1260.out‚0.046875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 714.96484375—ˆÃ’õ‚(:‡ íŸèÔ³¼ðß gpu.0.gpu‚38.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚47.0 gpu.0.powerWatts‚214.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.8578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ìŠkö‚(:à üŸèÔݸâÝ disk.sda1.in‚0 disk.nvme0n1.in‚ 775.22265625 disk.sda3.out‚ 11.69140625$ memory_percent‚90.84446430206299) proc.memory.availableMB‚ 1500.04296875* proc.memory.percent‚19.643259048461914 network.sent‚317272% disk./.usageGB‚1161.8771133422852 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.05078125$ disk.nvme0n1.out‚7852.3349609375 cpu‚23.311436250112987 proc.cpu.threads‚17 network.recv‚176623) disk./.usagePercent‚77.97454795623247 disk.sda.out‚ 11.69140625 disk.sda1.out‚0 disk.dm-0.in‚ 775.2265625" proc.memory.rssMB‚ 3218.3515625" disk.nvme0c0n1.in‚ 775.22265625& disk.nvme0c0n1.out‚7852.3349609375 disk.dm-1.in‚0 disk.dm-1.out‚ 11.69140625 disk.sda2.in‚0 disk.sda2.out‚0! disk.dm-0.out‚7852.3349609375 disk.sda3.in‚0 disk.sda.in‚00Zp ÷‚(:’ üŸèÔÕ‘ÿÝ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.995' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.39900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02b#_ø‚(:Ô ‹ èÔ¶ÿ¸à proc.cpu.threads‚17 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 834.14453125 disk.sda.in‚0 disk.nvme0n1.in‚ 834.14453125) proc.memory.availableMB‚ 1499.76171875 network.sent‚322745) disk./.usagePercent‚77.97454437212699% disk./.usageGB‚1161.8770599365234 disk.sda1.in‚0$ disk.nvme0n1.out‚8094.1279296875$ memory_percent‚90.84618091583252 cpu‚23.38348803652983 disk.drbd1260.out‚ 0.05078125 network.recv‚180792& disk.nvme0c0n1.out‚8094.1279296875 disk.sda.out‚12.0625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚12.0625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 834.14453125! disk.dm-0.out‚8094.1279296875 disk.sda3.in‚0 disk.sda3.out‚12.0625" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914U1õ±žù‚(:“ ‹ èÔ÷ïŸâ gpu.0.gpu‚88.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚43.0 gpu.0.powerWatts‚246.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.376999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 “ú‚(:ˆ š èÔà×ÞÝ gpu.0.gpu‚74.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚48.0 gpu.0.powerWatts‚325.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.0874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úVm5fû‚(:Û š èÔÀ– Þ disk.dm-1.out‚ 12.5859375 disk.drbd1260.out‚ 0.0546875 disk.sda2.out‚0 disk.dm-0.in‚ 834.14453125 disk.sda3.in‚0$ memory_percent‚90.84634780883789% disk./.usageGB‚1161.7769393920898 disk.dm-1.in‚0 disk.drbd1260.in‚0! disk.dm-0.out‚8382.9462890625$ disk.nvme0n1.out‚8382.9462890625 disk.sda3.out‚ 12.5859375 disk.sda.in‚0 disk.sda.out‚ 12.5859375 network.sent‚328320 network.recv‚184650) disk./.usagePercent‚77.96782519838331 disk.nvme0n1.in‚ 834.14453125& disk.nvme0c0n1.out‚8382.9462890625 disk.sda1.out‚0* proc.memory.percent‚19.643259048461914 cpu‚23.448713805873563 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 834.14453125 disk.sda1.in‚0' proc.memory.availableMB‚ 1499.734375" proc.memory.rssMB‚ 3218.3515625 proc.cpu.threads‚17Ãó“ü‚(:ˆ © èÔ³ÛùÜ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚46.0 gpu.0.powerWatts‚223.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.7258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ò–Xý‚(:Í © èÔ“üŒß network.sent‚336667 network.recv‚189600) disk./.usagePercent‚77.96782750245112% disk./.usageGB‚1161.7769737243652 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚13.25 disk.drbd1260.out‚ 0.0546875 disk.sda2.in‚0 disk.nvme0n1.in‚ 834.14453125" disk.nvme0c0n1.in‚ 834.14453125$ memory_percent‚90.84937572479248 disk.sda.in‚0 disk.sda.out‚13.25 disk.dm-1.out‚13.25 disk.dm-0.in‚ 834.14453125 disk.sda3.in‚0& disk.nvme0c0n1.out‚8767.7275390625) proc.memory.availableMB‚ 1499.23828125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.out‚0! disk.dm-0.out‚8767.7275390625$ disk.nvme0n1.out‚8767.7275390625" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914 cpu‚23.50591621488155©†jþ‚(:ß ¸ èÔšÅÿÞ network.sent‚340835 disk.sda1.in‚0 disk.drbd1260.out‚0.0625 disk.dm-0.in‚ 834.14453125% disk.nvme0n1.out‚9121.99755859375 disk.sda3.out‚ 13.64453125$ memory_percent‚90.84641933441162" proc.memory.rssMB‚ 3218.3515625% disk./.usageGB‚1161.9344749450684 disk.sda.in‚0 disk.sda1.out‚0 cpu‚23.56277935660425 proc.cpu.threads‚17) disk./.usagePercent‚77.97839754152484" disk.nvme0c0n1.in‚ 834.14453125" disk.dm-0.out‚9121.99755859375 disk.nvme0n1.in‚ 834.14453125 disk.sda3.in‚0) proc.memory.availableMB‚ 1499.72265625* proc.memory.percent‚19.643259048461914 disk.drbd1260.in‚0 network.recv‚193622' disk.nvme0c0n1.out‚9121.99755859375 disk.sda.out‚ 13.64453125 disk.dm-1.in‚0 disk.dm-1.out‚ 13.64453125 disk.sda2.in‚0 disk.sda2.out‚0)c]œÿ‚(:‘ ¸ èÔëÁÜß gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚40.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»‰^€‚(:Ó Ç èÔÊÝìÞ) disk./.usagePercent‚77.97839754152484" disk.dm-0.out‚9458.77490234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 14.05078125% disk./.usageGB‚1161.9344749450684 disk.sda3.in‚0 disk.sda3.out‚ 14.05078125 disk.dm-1.in‚0$ memory_percent‚90.84489345550537" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914 proc.cpu.threads‚17 network.recv‚197505 disk.nvme0n1.in‚948.0625% disk.nvme0n1.out‚9458.77490234375 disk.sda.out‚ 14.05078125 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1499.97265625 network.sent‚345591 disk.dm-0.in‚948.0625 disk.nvme0c0n1.in‚948.0625' disk.nvme0c0n1.out‚9458.77490234375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.0625 cpu‚23.61728086958057ÿQ3V“‚(:ˆ Ç èÔ÷›¸ß gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚45.0 gpu.0.powerWatts‚236.991' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.3982 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š„Š0‚+ _wandbruntime‚443  _runtime‚443 ¹ Ń¿ ' lr-AdamW/pg2‚0.00025004036554623064 trainer/global_step‚149 % _timestamp‚1.7884816160396788e+09 _runtime‚ 443.582067112 _step‚8 ' lr-AdamW/pg1‚0.00025004036554623064BGù=É„Ã _step‚8 ' lr-AdamW/pg1‚0.00025004036554623064 ' lr-AdamW/pg2‚0.00025004036554623064 trainer/global_step‚149 % _timestamp‚1.7884816160396788e+09 _runtime‚ 443.582067112ð(šO0…+ _wandbruntime‚443  _runtime‚443d{>¯Ü†ÂO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 447570099712 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2110875 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚835378 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1684502016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚93459456 XEDeviceStatsMonitor.on_train_batch_start/requested_byü@µÖì$tes.all.allocated‚45794946136664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45344397588560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57948504064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9542685896704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4550819840 _runtime‚ 443.586269363 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 _step‚9 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚835375 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47274128755200 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3644947 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5306 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3644947 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47274103327232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3048105 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2170 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 122557440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47721673426944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45344423016528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 450523120136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3048105 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚819 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1306 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5306 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47274103327232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3048111 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10122774588928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 447570099712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57948504064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4550819840 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47721673426944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47274128755200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10122652031488 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580059594240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45791353877680 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3669737 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6717848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2110056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451138267648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1644 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451138267648 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6693052 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3048111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5306 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3048105 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2170 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 122557440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47721673426944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45344423016528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 450523120136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3048105 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚819 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1306 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5306 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47274103327232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3048111 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10122774588928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 447570099712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57948504064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4550819840 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47721673426944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47274128755200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10122652031488 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580059594240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45791353877680 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3669737 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6717848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2110056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451138267648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1644 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451138267648 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6693052 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3048111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5306 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10188163693568 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3694055 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48042852639232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57948504064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47588719771648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5306 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3068355 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9604231772672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45646068585760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4550819840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6762416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 454132867584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2124744 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2123925 _step‚10 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6762416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47588694343680 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 450564686336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47588694343680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4550819840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3694055 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 583811473920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3669239 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1306 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3068355 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 449946801734 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48042852639232 trainer/global_step‚149 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚840862 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5306 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5306 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 450564686336 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10188043246592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3068361 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 120446976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349Å…ØÉ! ŠÒM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3068361 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 120446976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9604260870656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10188163693568 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3694055 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48042852639232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57948504064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47588719771648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5306 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3068355 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9604231772672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45646068585760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4550819840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6762416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 454132867584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2124744 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2123925 _step‚10 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6762416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47588694343680 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 450564686336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47588694343680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4550819840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3694055 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 583811473920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3669239 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1306 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3068355 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 449946801734 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48042852639232 trainer/global_step‚149 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚840862 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5306 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5306 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 450564686336 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10188043246592 cìhj0‹+ _wandbruntime‚445  _runtime‚445?ZSŒÍ .loss/FRt1/detection_loss‚3.263168672213749 2loss/SRqqt4/assignment_loss‚3.4182797510912466 /loss/FRt4/assignment_loss‚7.217275462019334 /loss/FRt4/detection_loss‚2.6113230128251357 /loss/FBt1/detection_loss‚11.812040102023344 _runtime‚ 445.919929648 .loss/FBt3/detection_loss‚5.377685683426158 /loss/FBt4/detection_loss‚10.758858828743298 /loss/FRt2/assignment_loss‚7.315704924161317 0loss/FBt4/assignment_loss‚0.4432798734497434 1loss/SRqqt1/detection_loss‚2.0164054437368004 1loss/SRqqt4/detection_loss‚1.3304324736470168 /loss/FBt2/detection_loss‚1.4932615097979627 /loss/FRt1/assignment_loss‚7.441604521809792 _step‚11 /loss/FRt3/assignment_loss‚7.155303120613098 1loss/SRqqt2/assignment_loss‚3.292937704495021 0loss/FBt1/assignment_loss‚0.6673782788745414 0loss/FBt2/assignment_loss‚0.5664338157762452 1loss/FBt3/assignment_loss‚0.34975231175886967 trainer/global_step‚149 &loss/total_loss‚101.44752872502858 .loss/FRt2/detection_loss‚2.513254558148443 1loss/SRqqt3/assignment_loss‚3.404682108000213 1loss/SRqqt2/detection_loss‚5.7308346671717505 2loss/SRqqt1/assignment_loss‚3.3831254641215005 /loss/FRt3/detection_loss‚3.4996297359466553 0loss/SRqqt3/detection_loss‚6.384876701177335 epoch‚0 $ _timestamp‚1.788481618378616e+09O™áèWÑ 0loss/SRqqt3/detection_loss‚6.384876701177335 epoch‚0 $ _timestamp‚1.788481618378616e+09 .loss/FRt1/detection_loss‚3.263168672213749 2loss/SRqqt4/assignment_loss‚3.4182797510912466 /loss/FRt4/assignment_loss‚7.217275462019334 /loss/FRt4/detection_loss‚2.6113230128251357 /loss/FBt1/detection_loss‚11.812040102023344 _runtime‚ 445.919929648 .loss/FBt3/detection_loss‚5.377685683426158 /loss/FBt4/detection_loss‚10.758858828743298 /loss/FRt2/assignment_loss‚7.315704924161317 0loss/FBt4/assignment_loss‚0.4432798734497434 1loss/SRqqt1/detection_loss‚2.0164054437368004 1loss/SRqqt4/detection_loss‚1.3304324736470168 /loss/FBt2/detection_loss‚1.4932615097979627 /loss/FRt1/assignment_loss‚7.441604521809792 _step‚11 /loss/FRt3/assignment_loss‚7.155303120613098 1loss/SRqqt2/assignment_loss‚3.292937704495021 0loss/FBt1/assignment_loss‚0.6673782788745414 0loss/FBt2/assignment_loss‚0.5664338157762452 1loss/FBt3/assignment_loss‚0.34975231175886967 trainer/global_step‚149 &loss/total_loss‚101.44752872502858 .loss/FRt2/detection_loss‚2.513254558148443 1loss/SRqqt3/assignment_loss‚3.404682108000213 1loss/SRqqt2/detection_loss‚5.7308346671717505 2loss/SRqqt1/assignment_loss‚3.3831254641215005 /loss/FRt3/detection_loss‚3.4996297359466553 `¿äV]Ž‚(:Ò Ö èÔËÏÕß cpu‚23.667152001837156 proc.cpu.threads‚17 disk.nvme0c0n1.in‚948.0625% disk.nvme0c0n1.out‚9657.708984375 disk.dm-1.in‚0 disk.dm-1.out‚ 14.64453125 disk.dm-0.out‚9657.708984375$ memory_percent‚90.84813594818115* proc.memory.percent‚19.643259048461914) disk./.usagePercent‚77.97871857497277% disk./.usageGB‚1161.9392585754395 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 14.64453125 network.recv‚201319 disk.sda.in‚0 disk.sda.out‚ 14.64453125 disk.drbd1260.in‚0 disk.nvme0n1.in‚948.0625 disk.sda3.in‚0) proc.memory.availableMB‚ 1499.44140625" proc.memory.rssMB‚ 3218.3515625 network.sent‚350945 disk.sda1.out‚0 disk.drbd1260.out‚ 0.06640625 disk.dm-0.in‚948.0625# disk.nvme0n1.out‚9657.708984375ˆÖð’‚(:‡ Ö èÔÿà‘á gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚45.0 gpu.0.powerWatts‚191.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚005i]‚(:Ò å èÔ©òŒÞ disk.sda2.in‚0" disk.dm-0.out‚9825.20458984375 disk.sda3.in‚0 disk.nvme0c0n1.in‚948.0625( proc.memory.availableMB‚ 1499.8828125 network.recv‚207836) disk./.usagePercent‚77.97871908698784$ disk./.usageGB‚1161.939266204834 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.06640625 disk.dm-0.in‚948.0625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 15.0703125 disk.sda2.out‚0 disk.nvme0n1.in‚948.0625% disk.nvme0n1.out‚9825.20458984375' disk.nvme0c0n1.out‚9825.20458984375 disk.sda.out‚ 15.0703125# memory_percent‚90.8454418182373 cpu‚23.712622688793335 network.sent‚401356 disk.sda3.out‚ 15.0703125 disk.sda1.in‚0" proc.memory.rssMB‚ 3218.3515625* proc.memory.percent‚19.643259048461914 proc.cpu.threads‚17 disk.dm-1.in‚0 ~ “‘‚(:ˆ å èÔìâ¬à gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚43.0 gpu.0.powerWatts‚134.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.9976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#6X’‚(:Í ô èÔ›íæß* proc.memory.percent‚19.643259048461914) disk./.usagePercent‚77.98133317991997% disk./.usageGB‚1161.9782180786133# disk.dm-0.out‚10364.40185546875& disk.nvme0n1.out‚10364.40185546875( disk.nvme0c0n1.out‚10364.40185546875 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚23.754445897323023 network.sent‚407617 disk.sda3.in‚0 disk.sda3.out‚15.4375 disk.sda.in‚0) proc.memory.availableMB‚ 1499.84765625 proc.cpu.threads‚17 network.recv‚212627 disk.sda1.in‚0 disk.dm-1.out‚15.4375 disk.drbd1260.out‚ 0.0703125 disk.sda2.in‚0" proc.memory.rssMB‚ 3218.3515625 disk.dm-0.in‚948.0625 disk.nvme0n1.in‚948.0625 disk.nvme0c0n1.in‚948.0625 disk.sda.out‚15.4375 disk.sda1.out‚0 disk.drbd1260.in‚0# memory_percent‚90.8456563949585’3|ž“‚(:“ ô èÔ⼶à gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚44.0 gpu.0.powerWatts‚243.426' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.685199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÖ]Km”‚(:â ƒ¡èÔ¦Ú¹á disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚23.79407711436081# disk.dm-0.out‚10590.83544921875 disk.sda.out‚ 15.984375 disk.dm-1.out‚ 15.984375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0703125$ memory_percent‚90.84422588348389* proc.memory.percent‚19.643282890319824 proc.cpu.threads‚17 network.sent‚413077 network.recv‚216393) disk./.usagePercent‚77.98710128567393 disk.sda2.out‚0! disk.nvme0n1.in‚ 1074.26953125 disk.sda3.out‚ 15.984375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1074.26953125& disk.nvme0n1.out‚10590.83544921875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1074.26953125( disk.nvme0c0n1.out‚10590.83544921875) proc.memory.availableMB‚ 1500.08203125# proc.memory.rssMB‚ 3218.35546875$ disk./.usageGB‚1162.064167022705 disk.sda2.in‚0>™Dñ•‚(:’ ƒ¡èÔö׊á gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚44.0 gpu.0.powerWatts‚234.864' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚46.97280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬žò˜q–‚(:æ ’¡èÔ‘°ÞÝ disk.sda2.out‚0( disk.nvme0c0n1.out‚11033.81201171875 disk.sda.out‚ 16.3828125 disk.sda1.in‚0 disk.dm-1.out‚ 16.3828125) disk./.usagePercent‚77.99549014055593$ disk./.usageGB‚1162.189167022705 disk.sda3.in‚0 disk.sda3.out‚ 16.3828125# disk.nvme0c0n1.in‚ 1074.26953125 disk.dm-1.in‚0 network.sent‚418515 network.recv‚220294 disk.drbd1260.out‚ 0.07421875# disk.dm-0.out‚11033.81201171875# memory_percent‚90.8442735671997) proc.memory.availableMB‚ 1500.07421875# proc.memory.rssMB‚ 3218.35546875* proc.memory.percent‚19.643282890319824 disk.drbd1260.in‚0 disk.dm-0.in‚ 1074.26953125! disk.nvme0n1.in‚ 1074.26953125& disk.nvme0n1.out‚11033.81201171875 disk.sda.in‚0 disk.sda1.out‚0 cpu‚23.832549269742955 proc.cpu.threads‚17 disk.sda2.in‚0àÀ™¡“—‚(:ˆ ’¡èÔ˜¨ýÞ gpu.0.gpu‚34.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚41.0 gpu.0.powerWatts‚117.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,d(fq˜‚(:æ ¡¡èÔ¼òß disk.sda.out‚ 16.8671875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.07421875 disk.sda2.out‚0 disk.dm-0.in‚ 1074.26953125& disk.nvme0n1.out‚11331.30029296875# proc.memory.rssMB‚ 3218.35546875 network.recv‚225205) disk./.usagePercent‚77.99549014055593# disk.nvme0c0n1.in‚ 1074.26953125 disk.sda3.out‚ 16.8671875$ memory_percent‚90.84429740905762* proc.memory.percent‚19.643282890319824 cpu‚23.868268160432418 proc.cpu.threads‚17$ disk./.usageGB‚1162.189167022705( disk.nvme0c0n1.out‚11331.30029296875 disk.sda1.out‚0 disk.dm-1.out‚ 16.8671875# disk.dm-0.out‚11331.30029296875! disk.nvme0n1.in‚ 1074.26953125 network.sent‚427107 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 1500.0703125¢÷8’™‚(:‡ ¡¡èÔÌ‚ß gpu.0.gpu‚99.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚48.0 gpu.0.powerWatts‚325.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÊ|¾oš‚(:ä °¡èÔ±æíÞ network.sent‚431980$ disk./.usageGB‚1162.439250946045# disk.nvme0c0n1.in‚ 1074.26953125 disk.sda.out‚ 17.23046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚11582.0673828125 disk.drbd1260.out‚0.078125 disk.dm-0.in‚ 1074.26953125! disk.nvme0n1.in‚ 1074.26953125% disk.nvme0n1.out‚11582.0673828125 disk.sda3.in‚0 disk.sda3.out‚ 17.23046875( proc.memory.availableMB‚ 1499.5703125) disk./.usagePercent‚78.01227348248564 disk.dm-1.out‚ 17.23046875 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.dm-0.out‚11582.0673828125$ memory_percent‚90.84734916687012* proc.memory.percent‚19.643282890319824 cpu‚23.901969006994594 network.recv‚230730 disk.sda.in‚0# proc.memory.rssMB‚ 3218.35546875 proc.cpu.threads‚17^\ÇÖ“›‚(:ˆ °¡èÔΖ¾ß gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚45.0 gpu.0.powerWatts‚242.036' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.4072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0My•jœ‚(:ß ¿¡èÔÕïæà disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.01069852413501 disk.dm-1.in‚0 disk.dm-1.out‚ 17.7890625 disk.dm-0.in‚ 1221.0703125" disk.dm-0.out‚11881.7314453125 disk.sda3.in‚0# memory_percent‚90.8461332321167) proc.memory.availableMB‚ 1499.76953125 network.sent‚437750 disk.sda3.out‚ 17.7890625 cpu‚23.933894139612875 proc.cpu.threads‚17' disk.nvme0c0n1.out‚11881.7314453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.078125 disk.nvme0n1.in‚ 1221.0703125% disk.nvme0n1.out‚11881.7314453125# proc.memory.rssMB‚ 3218.35546875* proc.memory.percent‚19.643282890319824 network.recv‚234898% disk./.usageGB‚1162.4157829284668" disk.nvme0c0n1.in‚ 1221.0703125 disk.sda.in‚0 disk.sda.out‚ 17.7890625`’Ø+ž‚(:“ ¿¡èÔ뇻á gpu.0.gpu‚61.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.773' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.354600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÉçfcž‚(:Ø ΡèÔ°¹¢ß$ disk.nvme0n1.out‚12347.873046875 disk.dm-1.in‚0 disk.dm-0.in‚ 1221.0703125 disk.sda3.out‚18.09375) proc.memory.availableMB‚ 1500.01953125* proc.memory.percent‚19.643282890319824 cpu‚23.965377283983393 disk.drbd1260.out‚ 0.08203125! disk.dm-0.out‚12347.873046875 disk.sda3.in‚0$ memory_percent‚90.84460735321045# proc.memory.rssMB‚ 3218.35546875 proc.cpu.threads‚17 network.sent‚442828 network.recv‚238561% disk./.usageGB‚1162.4157066345215" disk.nvme0c0n1.in‚ 1221.0703125& disk.nvme0c0n1.out‚12347.873046875 disk.sda.in‚0 disk.sda.out‚18.09375 disk.sda1.in‚0( disk./.usagePercent‚78.0106934039843 disk.sda1.out‚0 disk.dm-1.out‚18.09375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1221.0703125I;ßžŸ‚(:“ ΡèÔ™¥âà gpu.0.gpu‚53.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚41.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø dÓl ‚(:á Ý¡èÔšç«Þ) disk./.usagePercent‚78.04424907951979% disk./.usageGB‚1162.9157104492188 disk.drbd1260.out‚ 0.08203125 disk.nvme0n1.in‚ 1221.0703125 disk.sda3.out‚ 18.55859375 disk.sda.in‚0 disk.sda.out‚ 18.55859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚12848.576171875# proc.memory.rssMB‚ 3218.35546875* proc.memory.percent‚19.643282890319824 proc.cpu.threads‚17 network.recv‚244547 disk.dm-1.out‚ 18.55859375 disk.sda2.out‚0! disk.dm-0.out‚12848.576171875$ disk.nvme0n1.out‚12848.576171875" disk.nvme0c0n1.in‚ 1221.0703125( proc.memory.availableMB‚ 1499.5234375 cpu‚23.995384014317686 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1221.0703125 disk.sda3.in‚0$ memory_percent‚90.84763526916504 network.sent‚449831 Á©ßž¡‚(:“ Ý¡èÔãÁ‰à gpu.0.gpu‚84.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚45.0 gpu.0.powerWatts‚251.452' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.290400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„%”´0¢+ _wandbruntime‚589  _runtime‚589ã³ÅÄ£¾ % _timestamp‚1.7884817623102863e+09 _runtime‚ 589.851833087 _step‚12 & lr-AdamW/pg1‚0.0003339465284812073 & lr-AdamW/pg2‚0.0003339465284812073 trainer/global_step‚199êLȤ _step‚12 & lr-AdamW/pg1‚0.0003339465284812073 & lr-AdamW/pg2‚0.0003339465284812073 trainer/global_step‚199 % _timestamp‚1.7884817623102863e+09 _runtime‚ 589.851833087 þŠq0¥+ _wandbruntime‚589  _runtime‚589ÙÐݽ%¦ÈO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1306 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2170 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8923504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12619979695104 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 600075922844 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 597327259136 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4862899 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 772398399488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1699445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚93459456 % _timestamp‚1.7884817623116066e+09 _runtime‚ 589.853812808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4550819840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4550819840 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2170 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60427972876528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4060605 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1684502016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4060611 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1109578 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5306 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4060611 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63604575004672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63003679577600 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 597327259136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 596509091828 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5306 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60427947448560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63003654149632 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4862899 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚832 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13392284635136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5306 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63600981408768 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4887689 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1700274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 772304940032 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 600895427072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 600895427072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2170 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4060605 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63003679577600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚829 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5306 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018241024 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8948300 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61028048799372 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 122557440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 740720128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2809852 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2809020 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12620008793088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4887689 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63600981408768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57948504064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456804352 XEDeviceStatsMonitor.on_train_batch_starÐGt/allocated_bytes.all.allocated‚63604575004672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1644 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H_Ò'§ÌO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57948504064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13392407192576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4887689 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63600981408768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57948504064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456804352 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63604575004672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1644 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4550819840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4550819840 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2170 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60427972876528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4060605 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1684502016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4060611 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1109578 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5306 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4060611 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63604575004672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63003679577600 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 597327259136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 596509091828 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5306 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60427947448560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63003654149632 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4862899 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚832 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13392284635136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5306 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63600981408768 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4887689 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1700274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 772304940032 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 600895427072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 600895427072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2170 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4060605 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63003679577600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚829 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5306 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018241024 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8948300 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61028048799372 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 122557440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 740720128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2809852 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2809020 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12620008793088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63318270594048 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1684502016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63918567011840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4912007 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚820 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8992868 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13457810064896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚199 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2822891 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8968046 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2823714 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4080855 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1644 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5306 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1115059 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4550819840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5306 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4887191 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12681554669056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60729618445760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 603890027008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚823 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776159852544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4550819840 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61332710309394 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8992868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63318245166080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60729643873728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 599499604442 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57948504064 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1115062 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61329118050202 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8968046 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4080855 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12681525571072 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63922160621056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2170 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1306 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95543296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63318270594048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 599499604442 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57948504064 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1115062 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61329118050202 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8968046 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4080855 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12681525571072 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63922160621056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2170 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1306 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95543296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63318270594048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63318270594048 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1684502016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63918567011840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4912007 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚820 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8992868 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13457810064896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚199 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2822891 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8968046 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2823714 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4080855 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1644 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5306 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1115059 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4550819840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5306 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4887191 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12681554669056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60729618445760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 603890027008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚823 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776159852544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4550819840 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61332710309394 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8992868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63318245166080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60729643873728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 FœºlL¬Æ .loss/FBt2/detection_loss‚8.165041805057175 .loss/FBt4/detection_loss‚7.170417064114621 /loss/FRt3/assignment_loss‚6.880761384963989 1loss/SRqqt4/assignment_loss‚3.460384576661246 0loss/FBt3/assignment_loss‚0.4966150002377058 .loss/FRt3/detection_loss‚4.152219533920288 .loss/FRt4/detection_loss‚1.990480643717477 0loss/FBt2/assignment_loss‚0.5880122567821098 .loss/FRt1/detection_loss‚3.831800193562136 .loss/FBt3/detection_loss‚1.884129366997377 _step‚15 0loss/FBt1/assignment_loss‚0.4141616397167843 0loss/SRqqt4/detection_loss‚1.744509839644696 %loss/total_loss‚95.48229694057983 1loss/SRqqt3/assignment_loss‚1.710887706247709 1loss/SRqqt3/detection_loss‚3.3418005156901573 epoch‚0 trainer/global_step‚199 .loss/FRt2/detection_loss‚2.590363328263425 0loss/FRt1/assignment_loss‚7.3536380297177795 0loss/FRt2/assignment_loss‚7.3182716997046215 1loss/SRqqt1/assignment_loss‚3.360015117515952 1loss/FBt4/assignment_loss‚0.35425097511247977 0loss/SRqqt2/detection_loss‚4.161239483406846 % _timestamp‚1.7884817649725199e+09 _runtime‚ 592.517197019 /loss/FRt4/assignment_loss‚7.286820101912004 1loss/SRqqt2/assignment_loss‚3.194117331169021 0loss/SRqqt1/detection_loss‚5.312360773652287 .loss/FBt1/detection_loss‚8.719998572811935|;‰­Ê .loss/FRt2/detection_loss‚2.590363328263425 0loss/FRt1/assignment_loss‚7.3536380297177795 0loss/FRt2/assignment_loss‚7.3182716997046215 1loss/SRqqt1/assignment_loss‚3.360015117515952 1loss/FBt4/assignment_loss‚0.35425097511247977 0loss/SRqqt2/detection_loss‚4.161239483406846 % _timestamp‚1.7884817649725199e+09 _runtime‚ 592.517197019 /loss/FRt4/assignment_loss‚7.286820101912004 1loss/SRqqt2/assignment_loss‚3.194117331169021 0loss/SRqqt1/detection_loss‚5.312360773652287 .loss/FBt1/detection_loss‚8.719998572811935 .loss/FBt2/detection_loss‚8.165041805057175 .loss/FBt4/detection_loss‚TÞX¥Ç7.170417064114621 /loss/FRt3/assignment_loss‚6.880761384963989 1loss/SRqqt4/assignment_loss‚3.460384576661246 0loss/FBt3/assignment_loss‚0.4966150002377058 .loss/FRt3/detection_loss‚4.152219533920288 .loss/FRt4/detection_loss‚1.990480643717477 0loss/FBt2/assignment_loss‚0.5880122567821098 .loss/FRt1/detection_loss‚3.831800193562136 .loss/FBt3/detection_loss‚1.884129366997377 _step‚15 0loss/FBt1/assignment_loss‚0.4141616397167843 0loss/SRqqt4/detection_loss‚1.744509839644696 %loss/total_loss‚95.48229694057983 1loss/SRqqt3/assignment_loss‚1.710887706247709 1loss/SRqqt3/detection_loss‚3.3418005156901573 epoch‚0 trainer/global_step‚199ÓòY]®‚(:Ò ì¡èÔ¤¨ÏÞ network.sent‚455910" disk.nvme0c0n1.in‚ 1221.0703125' disk.nvme0c0n1.out‚13170.3193359375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0859375 disk.nvme0n1.in‚ 1221.0703125# proc.memory.rssMB‚ 3218.35546875 network.recv‚248982 disk.sda.out‚19.125 disk.sda2.out‚0 disk.dm-0.in‚ 1221.0703125" disk.dm-0.out‚13170.3193359375% disk.nvme0n1.out‚13170.3193359375 disk.sda3.in‚0 disk.sda3.out‚19.125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚90.84348678588867 cpu‚24.02359910844218 proc.cpu.threads‚17) disk./.usagePercent‚78.04582813399098% disk./.usageGB‚1162.9392395019531 disk.sda.in‚0 disk.dm-1.out‚19.125 disk.sda2.in‚0' proc.memory.availableMB‚ 1500.203125* proc.memory.percent‚19.6432828903198244>V㓯‚(:ˆ ì¡èÔ¢¦õÞ gpu.0.gpu‚92.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚40.0 gpu.0.powerWatts‚115.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”©Ýp°‚(:å û¡èÔ²ã¸ß disk.drbd1260.out‚ 0.0859375 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚505265) disk./.usagePercent‚78.04582813399098 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1341.96484375" disk.dm-0.out‚13397.8818359375! disk.nvme0n1.in‚ 1341.96484375 disk.sda3.out‚ 19.58203125 disk.sda.out‚ 19.58203125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚13397.8818359375' proc.memory.availableMB‚ 1501.015625* proc.memory.percent‚19.643282890319824 cpu‚24.050549713300413% disk./.usageGB‚1162.9392395019531# disk.nvme0c0n1.in‚ 1341.96484375' disk.nvme0c0n1.out‚13397.8818359375 disk.sda2.in‚0$ memory_percent‚90.83852767944336# proc.memory.rssMB‚ 3218.35546875 network.recv‚254591 disk.dm-1.out‚ 19.582031253zA“±‚(:ˆ û¡èÔ¤¼Ôß gpu.0.gpu‚66.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.10554504394531* gpu.0.memoryAllocatedBytes‚ 58502217728 gpu.0.temp‚41.0 gpu.0.powerWatts‚125.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»9½¹o²‚(:ä Š¢èÔÞ’›ß disk.dm-1.in‚0 disk.dm-1.out‚ 20.04296875 disk.sda2.in‚0) proc.memory.availableMB‚ 1500.75390625# disk.nvme0c0n1.in‚ 1395.34765625 disk.sda.out‚ 20.04296875 disk.sda1.in‚0" disk.dm-0.out‚13648.5693359375 disk.sda3.out‚ 20.04296875! proc.memory.rssMB‚ 3218.359375 proc.cpu.threads‚17 network.sent‚510329 disk.sda2.out‚0 disk.dm-0.in‚ 1395.34765625 disk.sda3.in‚0$ memory_percent‚90.84012508392334* proc.memory.percent‚19.643306732177734 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.08984375! disk.nvme0n1.in‚ 1395.34765625% disk.nvme0n1.out‚13648.5693359375 cpu‚24.07607290051765 network.recv‚257981) disk./.usagePercent‚78.04580304525263$ disk./.usageGB‚1162.938865661621' disk.nvme0c0n1.out‚13648.5693359375Vûz¦‘³‚(:† Š¢èÔð· à gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.10798645019531* gpu.0.memoryAllocatedBytes‚ 58504314880 gpu.0.temp‚39.0 gpu.0.powerWatts‚162.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ŸOj´‚(:ß ™¢èÔÀ‰ß# proc.memory.availableMB‚1500.75 cpu‚24.10087463130068 proc.cpu.threads‚17 network.sent‚516789 network.recv‚262766% disk./.usageGB‚1162.9185218811035# disk.nvme0c0n1.in‚ 1395.34765625 disk.sda.out‚ 20.4765625 disk.sda2.in‚0& disk.nvme0n1.out‚14007.57568359375 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 20.4765625$ memory_percent‚90.84014892578125! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 disk.drbd1260.out‚ 0.08984375( disk.nvme0c0n1.out‚14007.57568359375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1395.34765625# disk.dm-0.out‚14007.57568359375) disk./.usagePercent‚78.04443775707253 disk.dm-1.out‚ 20.4765625! disk.nvme0n1.in‚ 1395.34765625eز’µ‚(:‡ ™¢èÔ¯€à gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10798645019531* gpu.0.memoryAllocatedBytes‚ 58504314880 gpu.0.temp‚44.0 gpu.0.powerWatts‚229.205' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.841 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nöo¶‚(:ä ¨¢èԘʬÞ* proc.memory.percent‚19.643306732177734! disk.nvme0n1.in‚ 1395.34765625& disk.nvme0n1.out‚14402.73095703125 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1260.out‚0.09375 disk.sda2.out‚0 network.sent‚522688# disk.dm-0.out‚14402.73095703125 disk.sda.in‚0 disk.sda.out‚ 21.09765625 disk.drbd1260.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚78.0460114353855% disk./.usageGB‚1162.9419708251953# disk.nvme0c0n1.in‚ 1395.34765625( disk.nvme0c0n1.out‚14402.73095703125# memory_percent‚90.8402681350708) proc.memory.availableMB‚ 1500.73046875 cpu‚24.125723580632158 network.recv‚267328 disk.dm-0.in‚ 1395.34765625 disk.sda3.out‚ 21.09765625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 21.09765625! proc.memory.rssMB‚ 3218.359375Ë,ñ“·‚(:ˆ ¨¢èÔÚÆëß gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.10798645019531* gpu.0.memoryAllocatedBytes‚ 58504314880 gpu.0.temp‚41.0 gpu.0.powerWatts‚168.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚018-ìn¸‚(:ã ·¢èÔ¥Åºß disk.dm-1.in‚0 disk.dm-1.out‚ 21.6015625 disk.sda2.out‚0 disk.dm-0.in‚ 1395.34765625( disk.nvme0c0n1.out‚14653.22705078125! proc.memory.rssMB‚ 3218.359375 disk.sda1.out‚0! disk.nvme0n1.in‚ 1395.34765625& disk.nvme0n1.out‚14653.22705078125) proc.memory.availableMB‚ 1500.24609375* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17 network.sent‚527833 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 21.6015625$ memory_percent‚90.84322452545166 cpu‚24.149790688814196) disk./.usagePercent‚78.04443647703485% disk./.usageGB‚1162.9185028076172 disk.sda.out‚ 21.6015625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.09375 disk.sda2.in‚0# disk.dm-0.out‚14653.22705078125# disk.nvme0c0n1.in‚ 1395.34765625 network.recv‚270888 disk.sda.in‚0Þþuž¹‚(:“ ·¢èÔ’ÿŠà gpu.0.gpu‚63.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.10798645019531* gpu.0.memoryAllocatedBytes‚ 58504314880 gpu.0.temp‚42.0 gpu.0.powerWatts‚146.745' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.349000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îLtXhº‚(:Ý Æ¢èÔˆ¥´á disk.nvme0n1.in‚ 1513.4296875" disk.nvme0c0n1.in‚ 1513.4296875 disk.sda.in‚0! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 21.9453125$ memory_percent‚90.84172248840332( proc.memory.availableMB‚ 1500.4921875 disk.drbd1260.out‚ 0.09765625" disk.dm-0.out‚15072.1298828125% disk.nvme0n1.out‚15072.1298828125' disk.nvme0c0n1.out‚15072.1298828125 cpu‚24.17089864075001 proc.cpu.threads‚17% disk./.usageGB‚1162.9419708251953 disk.dm-0.in‚ 1513.4296875 disk.sda3.in‚0 disk.sda3.out‚ 21.9453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 21.9453125 network.sent‚533057 network.recv‚274541( disk./.usagePercent‚78.0460114353855‚ý½hœ»‚(:‘ Æ¢èÔ­¦öá gpu.0.gpu‚97.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚256.78' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.35599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q7ÚÜl¼‚(:á Õ¢èÔ¤®Üß" disk.dm-0.out‚15514.8330078125" disk.nvme0c0n1.in‚ 1513.4296875 disk.sda.in‚0 disk.sda.out‚ 22.48828125 disk.dm-1.in‚0 disk.dm-1.out‚ 22.48828125 disk.drbd1260.in‚0$ memory_percent‚90.84029197692871 network.recv‚279217% disk./.usageGB‚1162.9419708251953 disk.sda3.out‚ 22.48828125' disk.nvme0c0n1.out‚15514.8330078125 disk.sda2.in‚0( proc.memory.availableMB‚ 1500.7265625 cpu‚24.193030536207566 proc.cpu.threads‚17( disk./.usagePercent‚78.0460114353855 disk.nvme0n1.in‚ 1513.4296875 disk.sda3.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 network.sent‚539308 disk.dm-0.in‚ 1513.4296875% disk.nvme0n1.out‚15514.8330078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.09765625Omi‘½‚(:† Õ¢èÔ÷£­à gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚43.0 gpu.0.powerWatts‚240.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`»¿e¾‚(:Ú ä¢èÔ¡‘±Þ' disk.nvme0c0n1.out‚15750.3876953125 disk.sda.in‚0 disk.sda.out‚ 22.921875 disk.sda1.out‚0 disk.dm-1.out‚ 22.921875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1513.4296875 disk.dm-1.in‚0! proc.memory.rssMB‚ 3218.359375 cpu‚24.213048684395307 network.sent‚544233 disk.sda2.out‚0 disk.dm-0.in‚ 1513.4296875" disk.dm-0.out‚15750.3876953125" disk.nvme0c0n1.in‚ 1513.4296875 proc.cpu.threads‚17( disk./.usagePercent‚78.0460436923348 disk.drbd1260.out‚ 0.1015625% disk.nvme0n1.out‚15750.3876953125 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚90.84048271179199( proc.memory.availableMB‚ 1500.6953125* proc.memory.percent‚19.643306732177734 network.recv‚282909% disk./.usageGB‚1162.9424514770508 disk.sda3.out‚ 22.921875Êßô;“¿‚(:ˆ ä¢èÔ»˜»ß gpu.0.gpu‚79.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚137.736' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß–º_À‚(:Ô ó¢èÔ´Ý´á( disk./.usagePercent‚78.0460436923348 disk.sda3.out‚23.3125" disk.nvme0c0n1.in‚ 1513.4296875 disk.dm-1.out‚23.3125 cpu‚24.233287327329545 proc.cpu.threads‚17 network.sent‚551305 disk.dm-0.in‚ 1513.4296875 disk.sda3.in‚0 disk.sda.out‚23.3125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.1015625 disk.sda2.in‚0% disk./.usageGB‚1162.9424514770508" disk.dm-0.out‚15905.7822265625' disk.nvme0c0n1.out‚15905.7822265625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1513.4296875% disk.nvme0n1.out‚15905.7822265625$ memory_percent‚90.83895683288574( proc.memory.availableMB‚ 1500.9453125! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 network.recv‚286845î’¾ÓÁ‚(:’ ó¢èÔã gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚248.76' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.751999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª6¯×0Â+ _wandbruntime‚742  _runtime‚742lã–NÆÃÀ trainer/global_step‚249 % _timestamp‚1.7884819146034093e+09 _runtime‚ 742.145489819 _step‚16 ' lr-AdamW/pg1‚0.00041785269141618404 ' lr-AdamW/pg2‚0.00041785269141618404×”ïÊÄÄ _runtime‚ 742.145489819 _step‚16 ' lr-AdamW/pg1‚0.00041785269141618404 ' lr-AdamW/pg2‚0.00041785269141618404 trainer/global_step‚249 % _timestamp‚1.7884819146034093e+09б0Å+ _wandbruntime‚742  _runtime‚742¾|ÛÒ'ÆÌO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 964013954560 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76261151462080 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6080851 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15697302591488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126751744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6105641 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75511497308560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3507386 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1316 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1654 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76257559203096 _step‚17 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5073105 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761429504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 747084418560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 750652586496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 _runtime‚ 742.149265526 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15697273493504 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 746061894536 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79480289390592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79483882986496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 749628725552 % _timestamp‚1.7884819146068099e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5073105 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78733204972032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11178752 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6105641 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16661287448064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6080851 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2123611 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79480289390592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1383775 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78733230400000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5073111x ªMÈ ÇÐO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1383775 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78733230400000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5073111 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 964013954560 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76261151462080 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6080851 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15697302591488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126751744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6105641 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75511497308560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3507386 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1316 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1654 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76257559203096 _step‚17 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5073105 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761429504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 747084418560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 750652586496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 _runtime‚ 742.149265526 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15697273493504 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 746061894536 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79480289390592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79483882986496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 749628725552 % _timestamp‚1.7884819146068099e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5073105 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78733204972032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11178752 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6105641 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16661287448064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6080851 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2123611 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚794802893905921¾r0È+ _wandbruntime‚746  _runtime‚746ËâÜ&ÉÖM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76562220712910 trainer/global_step‚249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 750079005184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2132039 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16726799384576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126738432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75813193733728 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5093355 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79801468602880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761429504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 967853276672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11223320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11223320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11198498 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15758848467456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 749052407150 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 % _timestamp‚1.7884819187364705e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1389262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79801468602880 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5093361 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6129959 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 753647186432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1654 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5093361 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6105143 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 967950917120 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 750079005184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3521298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2132885 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75813168305760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 746.282536755 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6105143 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1703376384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79047821416448 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76565812972102 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308”àŸœà&ÊÚM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11223320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11223320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11198498 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15758848467456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 749052407150 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 % _timestamp‚1.7884819187364705e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1389262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79801468602880 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5093361 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6129959 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 753647186432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1654 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5093361 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6105143 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 967950917120 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 750079005184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3521298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2132885 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75813168305760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 746.282536755 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6105143 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1703376384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79047821416448 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76565812972102 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76562220712910 trainer/global_step‚249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 750079005184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2132039 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16726799384576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126738432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75813193733728 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5093355 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79801468602880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761429504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 967853276672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144ž£ù­0Ë+ _wandbruntime‚746  _runtime‚746äp3KÌÅ .loss/FBt3/detection_loss‚5.172102376547727 .loss/FBt4/detection_loss‚8.418055852254232 /loss/FRt3/assignment_loss‚6.633251614040798 /loss/FRt1/detection_loss‚2.3459999228361994 .loss/FBt2/detection_loss‚2.598205027449787 0loss/FBt1/assignment_loss‚0.3582989687909257 1loss/SRqqt1/detection_loss‚2.3127163760061187 _step‚19 0loss/FBt4/assignment_loss‚0.5666881522750676 .loss/FRt4/detection_loss‚2.661713646620177 %loss/total_loss‚83.57279707674911 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.130591033549791 0loss/SRqqt2/detection_loss‚2.771230541553694 1loss/SRqqt3/detection_loss‚3.6620841129468036 epoch‚0 _runtime‚ 746.2871016 % _timestamp‚1.7884819187444355e+09 2loss/SRqqt4/assignment_loss‚3.0008484135686824 0loss/FBt2/assignment_loss‚0.5067316915953501 /loss/FRt2/detection_loss‚1.9757091170176864 /loss/FRt3/detection_loss‚1.8814310497707791 ,loss/FBt1/detection_loss‚8.4107297481969 /loss/FRt1/assignment_loss‚6.663223993033171 1loss/SRqqt1/assignment_loss‚2.053956509121065 2loss/SRqqt3/assignment_loss‚1.7819196765239422 1loss/FBt3/assignment_loss‚0.30410570083115246 -loss/FRt2/assignment_loss‚6.6837903228071 1loss/SRqqt2/assignment_loss‚3.241992586666776 1loss/SRqqt4/detection_loss‚2.4374206427451783d(OÍÉ % _timestamp‚1.7884819187444355e+09 2loss/SRqqt4/assignment_loss‚3.0008484135686824 0loss/FBt2/assignment_loss‚0.5067316915953501 /loss/FRt2/detection_loss‚1.9757091170176864 /loss/FRt3/detection_loss‚1.8814310497707791 ,loss/FBt1/detection_loss‚8.4107297481969 /loss/FRt1/assignment_loss‚6.663223993033171 1loss/SRqqt1/assignment_loss‚2.053956509121065 2loss/SRqqt3/assignment_loss‚1.7819196765239422 1loss/FBt3/assignment_loss‚0.30410570083115246 -loss/FRt2/assignment_loss‚6.6837903228071 1loss/SRqqt2/assignment_loss‚3.241992586666776 1loss/SRqqt4/detection_loss‚2.4374206427451783 .loss/FBt3/detection_loss‚5.172102376547727 .loss/FBt4/detection_loss‚8.418055852254232 /loss/FRt3/assignment_loss‚6.633251614040798 /loss/FRt1/detection_loss‚2.3459999228361994 .loss/FBt2/detection_loss‚2.598205027449787 0loss/FBt1/assignment_loss‚0.3582989687909257 1loss/SRqqt1/detection_loss‚2.3127163760061187 _step‚19 0loss/FBt4/assignment_loss‚0.5666881522750676 .loss/FRt4/detection_loss‚2.661713646620177 %loss/total_loss‚83.57279707674911 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.130591033549791 0loss/SRqqt2/detection_loss‚2.771230541553694 1loss/SRqqt3/detection_loss‚3.6620841129468036 epoch‚0 _runtime‚ 746.2871016¡ã~΂(:… ‚£èÔꙢá gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚190.35' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.07 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G½*eÏ‚(:Ú ‚£èÔîä disk.drbd1260.in‚0 disk.dm-0.in‚ 1652.12890625" disk.dm-0.out‚16228.9052734375 disk.sda3.out‚ 23.734375 disk.sda1.out‚0$ memory_percent‚90.83938598632812# disk.nvme0c0n1.in‚ 1652.12890625' disk.nvme0c0n1.out‚16228.9052734375! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 cpu‚24.251251041620012 disk.dm-1.out‚ 23.734375 disk.sda2.in‚0! disk.nvme0n1.in‚ 1652.12890625 disk.sda.out‚ 23.734375 disk.sda1.in‚0$ proc.memory.availableMB‚1500.875 proc.cpu.threads‚17 network.sent‚567633) disk./.usagePercent‚78.04447487816499 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.10546875 disk.sda2.out‚0% disk.nvme0n1.out‚16228.9052734375 disk.sda3.in‚0 disk.sda.in‚0 network.recv‚290490$ disk./.usageGB‚1162.919075012207ÅZ‡ÓnЂ(:ã ‘£èÔóΗè cpu‚24.27155334290607 proc.cpu.threads‚17 disk.sda3.in‚0' disk.nvme0c0n1.out‚16647.5966796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 24.25390625 disk.drbd1260.in‚0" disk.dm-0.out‚16647.5966796875 network.recv‚296353# disk.nvme0c0n1.in‚ 1652.12890625 disk.sda.out‚ 24.25390625 disk.sda2.out‚0% disk.nvme0n1.out‚16647.5966796875$ memory_percent‚90.83948135375977! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 network.sent‚604579) disk./.usagePercent‚78.04604983651564 disk.sda1.out‚0 disk.drbd1260.out‚ 0.10546875! disk.nvme0n1.in‚ 1652.12890625% disk./.usageGB‚1162.9425430297852 disk.sda3.out‚ 24.25390625 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1652.12890625' proc.memory.availableMB‚ 1500.859375J·µÑ‚(:’ ‘£èÔä÷ê gpu.0.gpu‚59.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚206.723' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚41.34460000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÉ“iÒ‚(:Þ  £èÔ’™œâ network.sent‚612599 network.recv‚301103# disk.nvme0c0n1.in‚ 1652.12890625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 24.703125 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚0.109375 disk.sda2.out‚0 disk.dm-0.in‚ 1652.12890625# disk.dm-0.out‚17258.95068359375 cpu‚24.290527409748872) disk./.usagePercent‚78.04605009252316% disk./.usageGB‚1162.9425468444824 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1652.12890625& disk.nvme0n1.out‚17258.95068359375& proc.memory.availableMB‚ 1500.59375* proc.memory.percent‚19.643306732177734( disk.nvme0c0n1.out‚17258.95068359375 disk.sda.in‚0 disk.sda.out‚ 24.703125 disk.dm-1.out‚ 24.703125$ memory_percent‚90.84110260009766! proc.memory.rssMB‚ 3218.359375Øàé“Ó‚(:ˆ  £èԭɹâ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚251.451' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RùÑtkÔ‚(:à ¯£èÔõÓˆâ disk.dm-1.in‚0 disk.dm-1.out‚ 25.078125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1652.12890625# disk.dm-0.out‚17509.42333984375& disk.nvme0n1.out‚17509.42333984375 disk.sda3.out‚ 25.078125$ memory_percent‚90.83950519561768 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1652.12890625) proc.memory.availableMB‚ 1500.85546875! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 cpu‚24.308796109228588 network.recv‚304883) disk./.usagePercent‚78.04605162856838# disk.nvme0c0n1.in‚ 1652.12890625 disk.sda1.out‚0 disk.drbd1260.out‚0.109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚615827$ disk./.usageGB‚1162.942569732666( disk.nvme0c0n1.out‚17509.42333984375 disk.sda.out‚ 25.078125šeÛžÕ‚(:“ ¯£èÔœÖàá gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚235.363' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.072599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”çÖ‚(:’ ¾£èÔñ”°Þ gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚50.0 gpu.0.powerWatts‚332.377' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.47540000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—#iׂ(:Þ ¾£èÔôؽß disk.dm-0.in‚ 1784.63671875& disk.nvme0n1.out‚17857.57275390625( proc.memory.availableMB‚ 1500.3203125 network.sent‚620839( disk.nvme0c0n1.out‚17857.57275390625 disk.sda.out‚25.40625 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚24.32658505730265 proc.cpu.threads‚17 disk.sda3.out‚25.40625# disk.nvme0c0n1.in‚ 1784.63671875 disk.dm-1.out‚25.40625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1784.63671875! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 network.recv‚308380% disk./.usageGB‚1161.8803939819336 disk.sda.in‚0 disk.drbd1260.out‚ 0.11328125 disk.sda2.out‚0# disk.dm-0.out‚17857.57275390625$ memory_percent‚90.84277153015137) disk./.usagePercent‚77.97476812271191 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0ª(èuqØ‚(:æ Í£èÔ÷¦ß( disk.nvme0c0n1.out‚17952.91650390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1500.07421875 disk.sda.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 1784.63671875& disk.nvme0n1.out‚17952.91650390625! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 cpu‚24.3436452161101 proc.cpu.threads‚17 network.recv‚312899# disk.nvme0c0n1.in‚ 1784.63671875 disk.dm-1.in‚0 disk.dm-1.out‚ 25.87109375 disk.drbd1260.out‚ 0.11328125# memory_percent‚90.8442735671997% disk./.usageGB‚1161.8803939819336 disk.sda.out‚ 25.87109375 disk.sda2.in‚0# disk.dm-0.out‚17952.91650390625! disk.nvme0n1.in‚ 1784.63671875 disk.sda3.out‚ 25.87109375 network.sent‚627063) disk./.usagePercent‚77.97476812271191Û4_ªÙ‚(:’ Í£èÔôΉà gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚205.182' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚41.03639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²þñ—Ú‚(:Þ Ü£èÔßçÐà disk.sda1.in‚0# disk.dm-0.out‚18362.20166015625! disk.nvme0n1.in‚ 1784.63671875 disk.sda3.in‚0 cpu‚24.360335272200246 network.sent‚634820 disk.sda1.out‚0 disk.dm-1.out‚ 26.296875 disk.drbd1260.out‚ 0.1171875 disk.sda2.in‚0& disk.nvme0n1.out‚18362.20166015625 disk.sda3.out‚ 26.296875% proc.memory.availableMB‚ 1500.5625 neµDaxÒtwork.recv‚317233 disk.sda.in‚0 disk.dm-0.in‚ 1784.63671875$ memory_percent‚90.84129333496094* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17) disk./.usagePercent‚77.97476965875711# disk.nvme0c0n1.in‚ 1784.63671875 disk.sda.out‚ 26.296875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 3218.359375% disk./.usageGB‚1161.8804168701172( disk.nvme0c0n1.out‚18362.20166015625Åå毓Û‚(:ˆ Ü£èԥ˜á gpu.0.gpu‚74.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“žÙ¿iÜ‚(:Þ ë£èÔ¿ËñÞ disk.dm-1.out‚ 26.640625 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚18723.01025390625* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17 network.sent‚638250 disk.sda.in‚0 disk.drbd1260.out‚ 0.1171875 disk.dm-0.in‚ 1784.63671875$ memory_percent‚90.84153175354004( proc.memory.availableMB‚ 1500.5234375# disk.nvme0c0n1.in‚ 1784.63671875( disk.nvme0c0n1.out‚18723.01025390625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 26.640625 cpu‚24.37638865864529 network.recv‚320940# disk./.usageGB‚1161.88041305542 disk.sda.out‚ 26.640625 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1784.63671875& disk.nvme0n1.out‚18723.01025390625! proc.memory.rssMB‚ 3218.359375) disk./.usagePercent‚77.97476940274957 disk.sda1.out‚086yžÝ‚(:“ ë£èÔœÏá gpu.0.gpu‚79.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚42.0 gpu.0.powerWatts‚115.416' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.083199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ʋ‘&žÞ‚(:“ ú£èÔ¨á§á gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚245.009' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.001799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÞ¬÷iß‚(:Þ ú£èÔ•‹“â disk.sda3.in‚0 disk.sda.out‚ 26.9609375 disk.sda2.in‚0! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17 network.recv‚324235) disk./.usagePercent‚77.96805176505111 disk.dm-0.in‚ 1880.56640625 disk.sda3.out‚ 26.9609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 26.9609375% disk.nvme0n1.out‚18892.8173828125' disk.nvme0c0n1.out‚18892.8173828125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.12109375$ memory_percent‚90.84310531616211' proc.memory.availableMB‚ 1500.265625 cpu‚24.39219043326301 disk.sda.in‚0 network.sent‚643198# disk./.usageGB‚1161.78031539917" disk.dm-0.out‚18892.8173828125# disk.nvme0c0n1.in‚ 1880.56640625 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1880.56640625¸ºP0à+ _wandbruntime‚875  _runtime‚875;µ Ãá½ trainer/global_step‚299 $ _timestamp‚1.788482048147656e+09 _runtime‚ 875.690522115 _step‚20 & lr-AdamW/pg1‚0.0005017588543511607 & lr-AdamW/pg2‚0.0005017588543511607ƒ{üÇâÁ _step‚20 & lr-AdamW/pg1‚0.0005017588543511607 & lr-AdamW/pg2‚0.0005017588543511607 trainer/global_step‚299 $ _timestamp‚1.788482048147656e+09 _runtime‚ 875.690522115§ ë0ã+ _wandbruntime‚875  _runtime‚875¬î®*Ô'äÎO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1316 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761429504 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 895614697244 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94462755794432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1657978 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1654 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94462781222400 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95359597372416 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6085605 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7323593 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 _step‚21 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95363190968320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2550723 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95359597372416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90595047168560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2549883 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 899181528260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13409204 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90595072596528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 trainer/global_step‚299 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91490661865804 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18774596389888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1156294947840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94462781222400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1156390504448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1703376384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 896841577984 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6085605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774567291904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95363190968320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95556608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 % _timestamp‚1.7884820481573775e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6085611 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94462755794432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7323593 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1657975 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124654592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19930986894336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19930862239744 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1703376384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 896841577984 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6085605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774567291904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95363190968320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95556608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 % _timestamp‚1.7884820481573775e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6085611 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94462755794432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7323593 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1657975 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124654592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19930986894336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19930862239744 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6085605 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7323593 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 _step‚21 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95363190968320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2550723 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95359597372416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90595047168560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2549883 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 899181528260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13409204 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90595072596528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 trainer/global_step‚299 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91490661865804 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144zà\f0æ+ _wandbruntime‚877  _runtime‚877ùèš©e çØM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90896743593728 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7347911 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19996380948992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18836113167872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7347911 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1663462 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1703376384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 899836164608 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6105861 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1654 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 903404345856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1316 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18836142265856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13428950 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2559129 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4221764 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7323095 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95677182975488 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4222591 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95543296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6105861 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90896718165760 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 903404345856 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6105855 _runtime‚ 877.881169401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13453772 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94777372238848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1160238683136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1663459 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 124641280 % _timestamp‚1.7884820503386207e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all+7Œ³y.peak‚30637 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19996256307712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91798915634810 trainer/global_step‚299 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0vpéÆâ&èÜM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1663459 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 124641280 % _timestamp‚1.7884820503386207e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19996256307712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91798915634810 trainer/global_step‚299 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90896743593728 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7347911 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19996380948992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18836113167872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7347911 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1663462 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1703376384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 899836164608 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6105861 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1654 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 903404345856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1316 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18836142265856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13428950 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2559129 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4221764 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7323095 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95677182975488 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4222591 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95543296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6105861 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90896718165760 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 903404345856 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6105855 _runtime‚ 877.881169401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13453772 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94777372238848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1160238683136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886ªh^0é+ _wandbruntime‚877  _runtime‚877—BÇQêË .loss/FRt1/detection_loss‚4.452538315206766 /loss/FRt3/detection_loss‚2.6582150538993434 0loss/SRqqt3/detection_loss‚2.796135872900486 % _timestamp‚1.7884820503411276e+09 _runtime‚ 877.882604656 /loss/FBt1/detection_loss‚1.7446412479626798 /loss/FRt1/assignment_loss‚7.236773639917374 /loss/FRt2/assignment_loss‚7.173178267123094 2loss/SRqqt4/assignment_loss‚2.0096737832393288 0loss/FBt1/assignment_loss‚0.3106434455480121 /loss/FBt4/assignment_loss‚0.579365185890837 1loss/SRqqt1/detection_loss‚3.4377320087514818 _step‚23 /loss/FRt3/assignment_loss‚5.540041967879894 2loss/SRqqt3/assignment_loss‚2.4509974321722985 1loss/FBt3/assignment_loss‚0.31383834910866426 /loss/FRt4/detection_loss‚1.7502475945935774 1loss/SRqqt4/detection_loss‚1.6191607266664505 .loss/FRt2/detection_loss‚3.775857942317849 /loss/FRt4/assignment_loss‚4.991951420128001 .loss/FBt2/detection_loss‚3.798302943011125 trainer/global_step‚299 1loss/SRqqt2/assignment_loss‚2.980089512052415 /loss/FBt4/detection_loss‚3.4829935831458947 1loss/SRqqt2/detection_loss‚3.1289334576341172 %loss/total_loss‚74.76956364282665 1loss/SRqqt1/assignment_loss‚3.335854023694992 0loss/FBt2/assignment_loss‚0.5785107197440084 .loss/FBt3/detection_loss‚4.623887150237958 epoch‚0gç–UëÏ _step‚23 /loss/FRt3/assignment_loss‚5.540041967879894 2loss/SRqqt3/assignment_loss‚2.4509974321722985 1loss/FBt3/assignment_loss‚0.31383834910866426 /loss/FRt4/detection_loss‚1.7502475945935774 1loss/SRqqt4/detection_loss‚1.6191607266664505 .loss/FRt2/detection_loss‚3.775857942317849 /loss/FRt4/assignment_loss‚4.991951420128001 .loss/FBt2/detection_loss‚3.798302943011125 trainer/global_step‚299 1loss/SRqqt2/assignment_loss‚2.980089512052415 /loss/FBt4/detection_loss‚3.4829935831458947 1loss/SRqqt2/detection_loss‚3.1289334576341172 %loss/total_loss‚74.76956364282665 1loss/SRqqt1/assignment_loss‚3.335854023694992 0loss/FBt2/assignment_loss‚0.5785107197440084 .loss/FBt3/detection_loss‚4.623887150237958 epoch‚0 .loss/FRt1/detection_loss‚4.452538315206766 /loss/FRt3/detection_loss‚2.6582150538993434 0loss/SRqqt3/detection_loss‚2.796135872900486 % _timestamp‚1.7884820503411276e+09 _runtime‚ 877.882604656 /loss/FBt1/detection_loss‚1.7446412479626798 /loss/FRt1/assignment_loss‚7.236773639917374 /loss/FRt2/assignment_loss‚7.173178267123094 2loss/SRqqt4/assignment_loss‚2.0096737832393288 0loss/FBt1/assignment_loss‚0.3106434455480121 /loss/FBt4/assignment_loss‚0.579365185890837 1loss/SRqqt1/detection_loss‚3.4377320087514818úùÕbì‚(:× ‰¤èÔëáÐÞ disk.sda3.in‚0$ memory_percent‚90.84196090698242! proc.memory.rssMB‚ 3218.359375 proc.cpu.threads‚17 disk.drbd1260.in‚0$ disk./.usageGB‚1161.780418395996' disk.nvme0c0n1.out‚19277.6181640625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚27.5625 disk.dm-0.in‚ 1880.56640625 network.recv‚328952) disk./.usagePercent‚77.96805867725453# disk.nvme0c0n1.in‚ 1880.56640625 disk.sda3.out‚27.5625' proc.memory.availableMB‚ 1500.453125* proc.memory.percent‚19.643306732177734 cpu‚24.405773199836926 network.sent‚649489 disk.sda.out‚27.5625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1880.56640625 disk.sda.in‚0 disk.drbd1260.out‚ 0.12109375" disk.dm-0.out‚19277.6181640625% disk.nvme0n1.out‚19277.6181640625*¥‘í‚(:† ‰¤èÔü¦Äß gpu.0.gpu‚47.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚50.0 gpu.0.powerWatts‚328.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Hênî‚(:ã ˜¤èÔû¾ÅÞ network.recv‚334262 disk.sda1.out‚0 disk.drbd1260.out‚0.125 disk.dm-0.in‚ 1880.56640625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1880.56640625* proc.memory.percent‚19.643306732177734 cpu‚24.419786807168542 network.sent‚699257) disk./.usagePercent‚77.96806200535248 disk.sda.in‚0 disk.dm-1.out‚ 28.07421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 28.07421875$ memory_percent‚90.84286689758301% disk./.usageGB‚1161.7804679870605 disk.sda.out‚ 28.07421875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1880.56640625( disk.nvme0c0n1.out‚19614.61181640625! proc.memory.rssMB‚ 3218.359375 disk.sda1.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚19614.61181640625& disk.nvme0n1.out‚19614.61181640625( proc.memory.availableMB‚ 1500.3046875 proc.cpu.threads‚17;`O““ï‚(:ˆ ˜¤èÔ‹ß gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚237.763' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<59nð‚(:ã §¤èÔÙÌÆÞ* proc.memory.percent‚19.643306732177734 network.recv‚337824) disk./.usagePercent‚77.97707807869337% disk./.usageGB‚1161.9148139953613 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1880.56640625 disk.sda.out‚ 28.44140625$ memory_percent‚90.84277153015137 disk.dm-1.out‚ 28.44140625 disk.dm-0.in‚ 1880.56640625# disk.dm-0.out‚19790.65869140625! disk.nvme0n1.in‚ 1880.56640625 disk.sda3.in‚0 disk.sda3.out‚ 28.44140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.125( disk.nvme0c0n1.out‚19790.65869140625( proc.memory.availableMB‚ 1500.3203125 cpu‚24.432866100299382 proc.cpu.threads‚17 network.sent‚704370 disk.dm-1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚19790.65869140625 disk.sda.in‚0! proc.memory.rssMB‚ 3218.359375OŸßé’ñ‚(:‡ §¤èÔ¤§öÞ gpu.0.gpu‚16.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×|4gjò‚(:ß ¶¤èÔöøß network.sent‚709386) disk./.usagePercent‚77.97863076438853 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚28.84375( proc.memory.availableMB‚ 1500.3203125* proc.memory.percent‚19.643306732177734 network.recv‚341534( disk.nvme0c0n1.out‚20134.82568359375 disk.sda.out‚28.84375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.12890625 disk.sda2.out‚0 disk.dm-0.in‚ 1979.80859375% disk./.usageGB‚1161.9379501342773# disk.nvme0c0n1.in‚ 1979.80859375 disk.sda.in‚0# disk.dm-0.out‚20133.32568359375 disk.sda3.in‚0 disk.sda3.out‚28.84375$ memory_percent‚90.84277153015137! proc.memory.rssMB‚ 3218.359375 disk.sda2.in‚0! disk.nvme0n1.in‚ 1979.80859375& disk.nvme0n1.out‚20135.32568359375 cpu‚24.447757935754446 proc.cpu.threads‚17¿ Y “ó‚(:ˆ ¶¤èÔ„œ¹ß gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚253.321' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.6642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ljÑžô‚(:“ ŤèÔሊß gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚255.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.040600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Ĭçmõ‚(:â ŤèÔì·ß network.sent‚715705 network.recv‚346332& disk.nvme0n1.out‚20618.26318359375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 2004.9609375 disk.dm-1.out‚ 29.44921875 disk.drbd1260.in‚0* proc.memory.percent‚19.643306732177734 disk.nvme0n1.in‚ 2004.9609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚24.461012289439026( disk.nvme0c0n1.out‚20618.26318359375) disk./.usagePercent‚77.97734355850636% disk./.usageGB‚1161.9187698364258# disk.dm-0.out‚20618.26318359375 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚90.84434509277344 disk.dm-0.in‚ 2004.9609375 disk.sda3.out‚ 29.44921875 disk.sda.in‚0 disk.sda.out‚ 29.44921875 disk.drbd1260.out‚ 0.12890625% proc.memory.availableMB‚ 1500.0625! proc.memory.rssMB‚ 3218.359375 proc.cpu.threads‚17Iÿ!Sö‚(:È Ô¤èÔƒÒ¸Þ network.sent‚721540 network.recv‚350360( disk./.usagePercent‚77.9789162127892 cpu‚24.474664204055905 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2004.9609375 disk.sda3.in‚0 disk.sda3.out‚30" disk.nvme0c0n1.in‚ 2004.9609375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚30 disk.drbd1260.in‚0 disk.dm-0.in‚ 2004.9609375& disk.nvme0n1.out‚20910.88916015625 disk.sda.in‚0 disk.sda.out‚30 disk.drbd1260.out‚ 0.1328125( proc.memory.availableMB‚ 1500.3046875! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734% disk./.usageGB‚1161.9422035217285# disk.dm-0.out‚20910.88916015625( disk.nvme0c0n1.out‚20910.88916015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.84286689758301¹×qž÷‚(:“ Ô¤èÔƒ’ºÞ gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚191.474' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.294799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷dlø‚(:á ã¤èÔÒ±á disk.drbd1260.in‚0# disk.dm-0.out‚21295.66259765625) disk./.usagePercent‚77.98312600068225 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.1328125 disk.sda2.out‚0& disk.nvme0n1.out‚21295.66259765625& proc.memory.availableMB‚ 1500.09375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0* proc.memory.percent‚19.643306732177734 cpu‚24.48774346693261 network.sent‚726673 disk.sda3.in‚0( disk.nvme0c0n1.out‚21295.66259765625 disk.sda1.in‚0 disk.dm-0.in‚ 2004.9609375 disk.nvme0n1.in‚ 2004.9609375$ memory_percent‚90.84415435791016! proc.memory.rssMB‚ 3218.359375 network.recv‚354152% disk./.usageGB‚1162.0049324035645 disk.sda3.out‚ 30.55859375" disk.nvme0c0n1.in‚ 2004.9609375 disk.sda.out‚ 30.55859375 disk.dm-1.out‚ 30.55859375üv^Œ‘ù‚(:† ã¤èÔ…ÛŽã gpu.0.gpu‚5.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õF÷kcú‚(:Ø ò¤èÔÿâãß disk.sda3.out‚30.96875$ memory_percent‚90.84115028381348 cpu‚24.50089278806539 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.13671875 disk.sda2.in‚0 disk.dm-0.in‚ 2004.9609375 disk.nvme0n1.in‚ 2004.9609375 disk.sda3.in‚0! proc.memory.rssMB‚ 3218.359375) disk./.usagePercent‚77.98731914808558 disk.sda.out‚30.96875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚30.96875 disk.sda2.out‚0% disk.nvme0n1.out‚21451.2607421875' disk.nvme0c0n1.out‚21451.2607421875 network.recv‚357670% disk./.usageGB‚1162.0674133300781 disk.sda.in‚0" disk.dm-0.out‚21451.2607421875" disk.nvme0c0n1.in‚ 2004.9609375( proc.memory.availableMB‚ 1500.5859375* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17 network.sent‚731716 disk.sda1.out‚0Âö”"“û‚(:ˆ ò¤èÔ€‘Èà gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚226.032' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.2064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ühü‚(:Ý ¥èÔë©™à) disk./.usagePercent‚77.99411512408952# disk.nvme0c0n1.in‚ 2129.02734375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚31.34375$ memory_percent‚90.84570407867432* proc.memory.percent‚19.643306732177734 cpu‚24.511274292776633' disk.nvme0c0n1.out‚21710.5732421875 disk.sda.out‚31.34375 disk.sda1.out‚0 disk.dm-1.out‚31.34375 disk.drbd1260.in‚0 disk.dm-0.in‚ 2129.02734375% disk.nvme0n1.out‚21710.5732421875) proc.memory.availableMB‚ 1499.83984375 network.sent‚737994 network.recv‚362126" disk.dm-0.out‚21710.5732421875! disk.nvme0n1.in‚ 2129.02734375 disk.sda3.in‚0% disk./.usageGB‚1162.1686782836914 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.13671875 disk.sda2.out‚0! proc.memory.rssMB‚ 3218.359375 proc.cpu.threads‚17Žõ‘ý‚(:† ¥èÔò”žà gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éï•Þ2þ- _wandbruntime‚1013  _runtime‚1013ÑH~ÄÅÿ¿ _step‚24 & lr-AdamW/pg1‚0.0005856650172861374 & lr-AdamW/pg2‚0.0005856650172861374 trainer/global_step‚349 % _timestamp‚1.7884821859785242e+09 _runtime‚1013.521092291«N¡É€Ã % _timestamp‚1.7884821859785242e+09 _runtime‚1013.521092291 _step‚24 & lr-AdamW/pg1‚0.0005856650172861374 & lr-AdamW/pg2‚0.0005856650172861374 trainer/global_step‚349éÜШ2- _wandbruntime‚1013  _runtime‚1013•mñë@‚áO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1932175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2976674 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2975826 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4908852 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚97653760 _runtime‚1013.526477813 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8516755 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111238905354240 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 _step‚25 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8541545 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8516755 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1316 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1348435135488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 trainer/global_step‚349 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23200422977536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1703376384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106727356787496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4908001 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111238905354240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1046598737408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1654 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111242498950144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111242498950144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105678622456528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 % _timestamp‚1.7884821859825623e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1046598737408 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7098105 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7098111 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7098105 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0HI÷úë'ƒåO SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8541545 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8516755 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1316 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1348435135488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 trainer/global_step‚349 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23200422977536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1703376384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106727356787496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4908001 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111238905354240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1046598737408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1654 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111242498950144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111242498950144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105678622456528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 % _timestamp‚1.7884821859825623e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1046598737408 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7098105 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7098111 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7098105 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1932175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2976674 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2975826 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4908852 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚97653760 _runtime‚1013.526477813 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8516755 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111238905354240 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 _step‚25åa›,2„- _wandbruntime‚1016  _runtime‚1016VW°`ï&…éM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15684224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1053161505280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15684224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 # _timestamp‚1.78848218912498e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8565863 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4922761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚97640448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1937659 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7118361 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1051724843790 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1049593324032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7118355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126738432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110506897633280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110506923061248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4921925 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2984266 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1049593324032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761429504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1053161505280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107028426038326 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105980293453728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8541047 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8541047 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2985099 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105980268025760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23265806434304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1053161505280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15684224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 # _timestamp‚1.78848218912498e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8565863 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4922761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚97640448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1937659 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7118361 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1051724843790 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1049593324032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7118355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126738432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110506897633280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110506923061248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4921925 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2984266 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1049593324032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761429504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1053161505280 S?DeviceStatsMonitor.on_trúµ‚Éain_batch_end/requested_bytes.all.freed‚107028426038326 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105980293453728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8541047 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8541047 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2985099 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105980268025760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23265806434304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136‘f2‡- _wandbruntime‚1016  _runtime‚1016¥öQˆË /loss/FRt2/assignment_loss‚6.393776660741762 2loss/SRqqt3/assignment_loss‚2.6178917315873234 1loss/FBt1/assignment_loss‚0.45857933091993136 0loss/FBt3/assignment_loss‚0.5650041909357302 /loss/FRt1/detection_loss‚3.3883382775345625 0loss/SRqqt3/detection_loss‚2.044102668536432 /loss/FRt3/assignment_loss‚5.603416241539849 1loss/SRqqt2/assignment_loss‚3.356511427462101 /loss/FBt1/detection_loss‚2.0851052166369644 .loss/FBt2/detection_loss‚3.262279236948756 _runtime‚1016.676189891 .loss/FRt2/detection_loss‚2.715665352898975 /loss/FRt1/assignment_loss‚6.606386174961012 2loss/SRqqt4/assignment_loss‚1.9434261686358776 1loss/SRqqt4/detection_loss‚1.4337408564867773 $ _timestamp‚1.788482189132971e+09 1loss/SRqqt1/assignment_loss‚3.353824131190777 /loss/FBt2/assignment_loss‚0.297612008518337 /loss/FBt4/detection_loss‚11.093851566314697 /loss/FRt4/assignment_loss‚4.508804914967114 .loss/FRt3/detection_loss‚2.434134047561222 .loss/FBt3/detection_loss‚3.723574321012239 epoch‚0 0loss/FBt4/assignment_loss‚1.1676580980420113 _step‚27 /loss/FRt4/detection_loss‚1.8727494138755545 1loss/SRqqt2/detection_loss‚2.7543894588947295 trainer/global_step‚349 1loss/SRqqt1/detection_loss‚3.8745997194200754 %loss/total_loss‚77.55542121562281@Ö3ÕU‰Ï _step‚27 /loss/FRt4/detection_loss‚1.8727494138755545 1loss/SRqqt2/detection_loss‚2.7543894588947295 trainer/global_step‚349 1loss/SRqqt1/detection_loss‚3.8745997194200754 %loss/total_loss‚77.55542121562281 /loss/FRt2/assignment_loss‚6.393776660741762 2loss/SRqqt3/assignment_loss‚2.6178917315873234 1loss/FBt1/assignment_loss‚0.45857933091993136 0loss/FBt3/assignment_loss‚0.5650041909357302 /loss/FRt1/detection_loss‚3.3883382775345625 0loss/SRqqt3/detection_loss‚2.044102668536432 /loss/FRt3/assignment_loss‚5.603416241539849 1loss/SRqqt2/assignment_loss‚3.356511427462101 /loss/FBt1/detection_loss‚2.0851052166369644 .loss/FBt2/detection_loss‚3.262279236948756 _runtime‚1016.676189891 .loss/FRt2/detection_loss‚2.715665352898975 /loss/FRt1/assignment_loss‚6.606386174961012 2loss/SRqqt4/assignment_loss‚1.9434261686358776 1loss/SRqqt4/detection_loss‚1.4337408564867773 $ _timestamp‚1.788482189132971e+09 1loss/SRqqt1/assignment_loss‚3.353824131190777 /loss/FBt2/assignment_loss‚0.297612008518337 /loss/FBt4/detection_loss‚11.093851566314697 /loss/FRt4/assignment_loss‚4.508804914967114 .loss/FRt3/detection_loss‚2.434134047561222 .loss/FBt3/detection_loss‚3.723574321012239 epoch‚0 0loss/FBt4/assignment_loss‚1.1676580980420113Û"ΈnŠ‚(:ã ¥èÔœÂÔÞ% disk./.usageGB‚1162.1926155090332 disk.dm-0.in‚ 2129.02734375" disk.dm-0.out‚22105.6787109375# disk.nvme0c0n1.in‚ 2129.02734375 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚90.84486961364746% disk.nvme0n1.out‚22105.6787109375 disk.sda3.in‚0 disk.sda3.out‚ 31.78515625' disk.nvme0c0n1.out‚22105.6787109375 disk.sda.out‚ 31.78515625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.140625* proc.memory.percent‚19.643306732177734! disk.nvme0n1.in‚ 2129.02734375 disk.dm-1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1499.9765625! proc.memory.rssMB‚ 3218.359375 cpu‚24.521461756031307 proc.cpu.threads‚17 network.sent‚743545 network.recv‚366126) disk./.usagePercent‚77.99572157136687 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 31.78515625„²i=‘‹‚(:† ¥èÔ×™Ôß gpu.0.gpu‚75.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚315.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.052 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0» l›Œ‚(: Ÿ¥èÔ©ÁÄÞ gpu.0.gpu‚3.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚319.663' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚63.93260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óñFh‚(:Ý Ÿ¥èÔÈ‹íÞ" disk.dm-0.out‚22523.4013671875% disk.nvme0n1.out‚22523.4013671875 disk.sda3.in‚0 disk.sda3.out‚ 32.1484375 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚90.84186553955078! proc.memory.rssMB‚ 3218.359375) disk./.usagePercent‚78.01249928113084% disk./.usageGB‚1162.4426155090332' disk.nvme0c0n1.out‚22523.4013671875 disk.dm-1.out‚ 32.1484375 disk.sda2.in‚0* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17 network.sent‚791918 network.recv‚371059 disk.dm-0.in‚ 2129.02734375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.140625& proc.memory.availableMB‚ 1500.46875! disk.nvme0n1.in‚ 2129.02734375# disk.nvme0c0n1.in‚ 2129.02734375 disk.sda.out‚ 32.1484375 disk.sda1.in‚0 cpu‚24.53196996092024e@ÖÄbŽ‚(:× ®¥èÔÁ°€ß disk.dm-0.out‚ 22967.171875! disk.nvme0n1.in‚ 2129.02734375! disk.nvme0n1.out‚ 22967.171875$ memory_percent‚90.84336757659912 network.recv‚374279) disk./.usagePercent‚78.01249928113084 disk.sda.in‚0) proc.memory.availableMB‚ 1500.22265625* proc.memory.percent‚19.643306732177734 cpu‚24.542421822050134 disk.sda3.out‚ 32.3671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 3218.359375 proc.cpu.threads‚17 network.sent‚796935% disk./.usageGB‚1162.4426155090332 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2129.02734375# disk.nvme0c0n1.out‚ 22967.171875 disk.sda.out‚ 32.3671875 disk.dm-1.out‚ 32.3671875 disk.drbd1260.out‚ 0.14453125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 2129.02734375 Òú*œ‚(:‘ ®¥èÔÜÁ¤à gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÝw€“‚(:ˆ ½¥èÔä¸Øß gpu.0.gpu‚95.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚226.296' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.2592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Ôhôe‘‚(:Ú ½¥èÔ·Óà disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2251.35546875$ disk.nvme0c0n1.out‚ 23492.9921875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0$ memory_percent‚90.84334373474121% disk./.usageGB‚1162.4427223205566 disk.dm-0.in‚ 2251.35546875 disk.dm-0.out‚ 23492.9921875! disk.nvme0n1.in‚ 2251.35546875 disk.sda.out‚ 32.96484375 disk.dm-1.out‚ 32.96484375 disk.sda2.out‚0( proc.memory.availableMB‚ 1500.2265625" disk.nvme0n1.out‚ 23492.9921875 disk.sda1.in‚0! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 proc.cpu.threads‚17 network.recv‚379017( disk./.usagePercent‚78.0125064493418 disk.sda3.out‚ 32.96484375 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.14453125 cpu‚24.55255036187735 network.sent‚805052å}зm’‚(:â Ì¥èÔå–ÏÞ disk.dm-1.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚23806.13818359375& disk.nvme0n1.out‚23806.13818359375# disk.nvme0c0n1.in‚ 2251.35546875 disk.sda1.in‚0$ memory_percent‚90.84188938140869! proc.memory.rssMB‚ 3218.359375* proc.memory.percent‚19.643306732177734 cpu‚24.563625206059104 network.sent‚808995 network.recv‚383328% disk./.usageGB‚1161.7891998291016! disk.nvme0n1.in‚ 2251.35546875 disk.sda3.in‚0( disk.nvme0c0n1.out‚23806.13818359375 disk.sda.in‚0 disk.sda.out‚ 33.515625) disk./.usagePercent‚77.96864800659836 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 33.515625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1484375) proc.memory.availableMB‚ 1500.46484375 disk.dm-0.in‚ 2251.35546875 disk.sda3.out‚ 33.515625 ¢ÕZ’“‚(:‡ Ì¥èÔò«çß gpu.0.gpu‚60.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚40.0 gpu.0.powerWatts‚114.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.871 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U•êot”‚(:é Û¥èÔÙ—øÝ network.sent‚814070 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 33.83203125 disk.drbd1260.in‚0& disk.nvme0n1.out‚23961.62060546875$ memory_percent‚90.84188938140869# proc.memory.rssMB‚ 3218.36328125) disk./.usagePercent‚77.96864928663602$ disk./.usageGB‚1161.789218902588# disk.nvme0c0n1.in‚ 2251.35546875 disk.dm-1.in‚0 disk.dm-0.in‚ 2251.35546875* proc.memory.percent‚19.643330574035645 cpu‚24.574160622771725( disk.nvme0c0n1.out‚23961.62060546875 disk.sda2.in‚0! disk.nvme0n1.in‚ 2251.35546875 disk.sda3.in‚0 disk.sda3.out‚ 33.83203125) proc.memory.availableMB‚ 1500.46484375 proc.cpu.threads‚17 network.recv‚386741 disk.sda.out‚ 33.83203125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.1484375 disk.sda2.out‚0# disk.dm-0.out‚23961.62060546875«å’4“•‚(:ˆ Û¥èÔÄž‡Þ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚229.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.8444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%• óq–‚(:æ ê¥èÔ¶’ÉÞ proc.cpu.threads‚17 network.recv‚390140# disk.nvme0c0n1.in‚ 2251.35546875 disk.sda.in‚0 disk.drbd1260.out‚ 0.15234375! disk.nvme0n1.in‚ 2251.35546875 disk.sda3.in‚0# memory_percent‚90.8419132232666# proc.memory.rssMB‚ 3218.36328125% disk./.usageGB‚1161.7892112731934' disk.nvme0c0n1.out‚24140.2646484375 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚24140.2646484375( proc.memory.availableMB‚ 1500.4609375 network.sent‚819171 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 34.30078125 disk.sda2.out‚0" disk.dm-0.out‚24140.2646484375* proc.memory.percent‚19.643330574035645 cpu‚24.585005513518297) disk./.usagePercent‚77.96864877462096 disk.sda.out‚ 34.30078125 disk.drbd1260.in‚0 disk.dm-0.in‚ 2251.35546875 disk.sda3.out‚ 34.30078125ØJ—‚(:’ ê¥èÔÝÊ€á gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚48.0 gpu.0.powerWatts‚336.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.27519999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G‡Wm˜‚(:â ù¥èÔîʹÞ% disk./.usageGB‚1161.7891731262207" disk.nvme0c0n1.in‚ 2416.6796875 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 34.7265625 proc.cpu.threads‚17 network.recv‚394857 disk.sda.out‚ 34.7265625 disk.sda2.out‚0$ memory_percent‚90.84203243255615* proc.memory.percent‚19.643378257751465' disk.nvme0c0n1.out‚24463.8935546875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.dm-0.out‚24463.8935546875 disk.nvme0n1.in‚ 2416.6796875% disk.nvme0n1.out‚24463.8935546875 disk.sda1.out‚0 disk.dm-1.out‚ 34.7265625 disk.drbd1260.out‚ 0.15234375 disk.dm-0.in‚ 2416.6796875 disk.sda3.in‚0) proc.memory.availableMB‚ 1500.44140625# proc.memory.rssMB‚ 3218.37109375 cpu‚24.595195275373765 network.sent‚825315) disk./.usagePercent‚77.96864621454563„‹Óâ’™‚(:‡ ù¥èÔÊÑ×ß gpu.0.gpu‚28.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚40.0 gpu.0.powerWatts‚115.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éiWªfš‚(:Û ˆ¦èÔ‹úÙÞ disk.sda3.in‚0& disk.nvme0c0n1.out‚24869.728515625 disk.dm-1.out‚ 35.234375# memory_percent‚90.8435583114624 network.sent‚831340% disk./.usageGB‚1161.7891731262207 disk.drbd1260.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1500.19140625# proc.memory.rssMB‚ 3218.37109375 cpu‚24.605734699042806 proc.cpu.threads‚17 disk.sda3.out‚ 35.234375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚19.643378257751465! disk.dm-0.out‚24869.728515625$ disk.nvme0n1.out‚24869.728515625" disk.nvme0c0n1.in‚ 2416.6796875 disk.sda.out‚ 35.234375 disk.drbd1260.out‚ 0.16015625 disk.sda2.out‚0 network.recv‚399329) disk./.usagePercent‚77.96864621454563 disk.dm-0.in‚ 2416.6796875 disk.nvme0n1.in‚ 2416.6796875|ÿé_“›‚(:ˆ ˆ¦èÔâÉ»à gpu.0.gpu‚74.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚193.579' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.7158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žt–žœ‚(:“ —¦èÔǪ§ß gpu.0.gpu‚85.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚195.537' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.107400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îaéMk‚(:à —¦èÔ¶†óß disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.16015625 disk.dm-0.in‚ 2416.6796875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 2416.6796875& disk.nvme0c0n1.out‚25120.353515625 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚836802 network.recv‚403371) disk./.usagePercent‚77.96865005465862 disk.sda2.in‚0! disk.dm-0.out‚25120.353515625 disk.sda.out‚ 35.64453125 disk.dm-1.out‚ 35.64453125# memory_percent‚90.8435583114624 disk.nvme0n1.in‚ 2416.6796875$ disk.nvme0n1.out‚25120.353515625 disk.sda3.out‚ 35.64453125 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚19.643378257751465 cpu‚24.61516570636697% disk./.usageGB‚1161.7892303466797 disk.sda2.out‚0) proc.memory.availableMB‚ 1500.19140625# proc.memory.rssMB‚ 3218.37109375 proc.cpu.threads‚17ùŠîÖ2ž- _wandbruntime‚1165  _runtime‚1165µDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1199924064768 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121960459450204 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127118213336064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125921882867200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1199924064768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24929183986688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125921857439232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 % _timestamp‚1.7884823381755128e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8110605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1196355896832 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1196355896832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121956867191220 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761429504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1165.719235606 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚852 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24929154888704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1540372942848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9759497 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127121806931968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125921882867200 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9759497 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120762172316528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018241024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8110611 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136P—æ|§£åO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1196355896832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121956867191220 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761429504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1165.719235606 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚852 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456804352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24929154888704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1540372942848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9759497 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127121806931968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125921882867200 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9759497 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120762172316528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018241024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8110611 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8110611 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26469430177792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9734707 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5607861 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1194720302660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127121806931968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2206375 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120762146888560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5607009 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3400634 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pM‡£™Dool.current‚ 53397684224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018241024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3401483 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26469556929536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 trainer/global_step‚399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2206378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17845312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1199924064768 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121960459450204 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127118213336064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125921882867200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1199924064768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24929183986688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125921857439232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 % _timestamp‚1.7884823381755128e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8110605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1196355896832 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637V§™2¤- _wandbruntime‚1168  _runtime‚1168ùÐ ñ&¥ëM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17914676 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17889854 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3409805 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1197710815274 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2211859 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17889854 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9758999 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127439392548352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3408970 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201277646498 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8130855 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1202918664704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8130861 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚838 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24990700764672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚835 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24990729862656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1544115868672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122265120960226 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9783815 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8130855 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121063817885760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26534941274624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 _runtime‚1168.530616973 _step‚30 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127435798939136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1199350483456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121063843313728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126236473883648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95543296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚399 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5620829 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761429504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1199350483456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127439392548352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2211862 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 124641280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122261528701034 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8130861 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172…œ*õ&¦ïM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17889854 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3409805 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1197710815274 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2211859 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17889854 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9758999 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127439392548352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3408970 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201277646498 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8130855 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1202918664704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8130861 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚838 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24990700764672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚835 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24990729862656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1544115868672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122265120960226 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9783815 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8130855 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121063817885760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26534941274624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 _runtime‚1168.530616973 _step‚30 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127435798939136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1199350483456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121063843313728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126236473883648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95543296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚399 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5620829 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761429504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1199350483456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127439392548352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2211862 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 124641280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122261528701034 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8130861 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17914676"ÐB2§- _wandbruntime‚1168  _runtime‚1168&_¦U¨Ï 1loss/FBt2/assignment_loss‚0.37966249137320063 1loss/FBt3/assignment_loss‚0.43251125749536284 0loss/FBt4/assignment_loss‚0.5437522886527909 /loss/FRt2/detection_loss‚2.3400081252934886 % _timestamp‚1.7884823409904532e+09 1loss/SRqqt1/detection_loss‚3.8929267610822404 1loss/SRqqt3/detection_loss‚2.7902556394601796 /loss/FBt2/detection_loss‚1.8282143105340891 .loss/FBt3/detection_loss‚4.445682244865518 .loss/FBt4/detection_loss‚4.649678495195177 1loss/SRqqt1/assignment_loss‚3.362443140574864 .loss/FRt4/detection_loss‚1.805555667053133 _step‚31 2loss/SRqqt3/assignment_loss‚2.3761901034937276 .loss/FRt3/detection_loss‚3.109318511753247 0loss/SRqqt2/detection_loss‚3.148229291703966 1loss/SRqqt4/detection_loss‚0.9976507819592952 trainer/global_step‚399 /loss/FRt3/assignment_loss‚5.897830247879028 /loss/FRt1/detection_loss‚3.4217723366760073 0loss/FRt4/assignment_loss‚3.0498174028908647 2loss/SRqqt4/assignment_loss‚1.8616878845095635 /loss/FBt1/detection_loss‚2.0504922107034163 _runtime‚1168.531784644 -loss/FRt1/assignment_loss‚6.6316343943278 2loss/SRqqt2/assignment_loss‚3.1331218004226686 %loss/total_loss‚66.93284189313057 epoch‚0 /loss/FRt2/assignment_loss‚4.374218691797817 1loss/FBt1/assignment_loss‚0.41018781343312644r­ãPY©Ó epoch‚0 /loss/FRt2/assignment_loss‚4.374218691797817 1loss/FBt1/assignment_loss‚0.41018781343312644 1loss/FBt2/assignment_loss‚0.37966249137320063 1loss/FBt3/assignment_loss‚0.43251125749536284 0loss/FBt4/assignment_loss‚0.5437522886527909 /loss/FRt2/detection_loss‚2.3400081252934886 % _timestamp‚1.7884823409904532e+09 1loss/SRqqt1/detection_loss‚3.8929267610822404 1loss/SRqqt3/detection_loss‚2.7902556394601796 /loss/FBt2/detection_loss‚1.8282143105340891 .loss/FBt3/detection_loss‚4.445682244865518 .loss/FBt4/detection_loss‚4.649678495195177 1loss/SRqqt1/assignment_loss‚3.362443140574864 .loss/FRt4/detection_loss‚1.805555667053133 _step‚31 2loss/SRqqt3/assignment_loss‚2.3761901034937276 .loss/FRt3/detection_loss‚3.109318511753247 0loss/SRqqt2/detection_loss‚3.148229291703966 1loss/SRqqt4/detection_loss‚0.9976507819592952 trainer/global_step‚399 /loss/FRt3/assignment_loss‚5.897830247879028 /loss/FRt1/detection_loss‚3.4217723366760073 0loss/FRt4/assignment_loss‚3.0498174028908647 2loss/SRqqt4/assignment_loss‚1.8616878845095635 /loss/FBt1/detection_loss‚2.0504922107034163 _runtime‚1168.531784644 -loss/FRt1/assignment_loss‚6.6316343943278 2loss/SRqqt2/assignment_loss‚3.1331218004226686 %loss/total_loss‚66.93284189313057Ǹ2hª‚(:Ý ¦¦èÔ—â¾Þ( disk./.usagePercent‚77.9686677191785 disk.nvme0n1.in‚ 2416.6796875" disk.nvme0c0n1.in‚ 2416.6796875 disk.dm-1.out‚ 35.88671875 disk.sda2.in‚0! disk.dm-0.out‚25276.044921875) proc.memory.availableMB‚ 1499.88671875# proc.memory.rssMB‚ 3218.37109375 network.recv‚406510$ disk./.usageGB‚1161.789493560791& disk.nvme0c0n1.out‚25276.044921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1640625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 35.88671875 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 2416.6796875# memory_percent‚90.8454179763794 cpu‚24.62447607666728 disk.sda1.out‚0 network.sent‚841681$ disk.nvme0n1.out‚25276.044921875 disk.sda3.in‚0 disk.sda.out‚ 35.88671875 disk.sda1.in‚0* proc.memory.percent‚19.643378257751465T¡”C’«‚(:‡ ¦¦èÔˆÈýÞ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚244.965' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S„j?h¬‚(:Ý µ¦èÔÔ£øÞ disk.drbd1260.out‚ 0.1640625 disk.sda2.in‚0 disk.sda3.in‚0# memory_percent‚90.8423900604248# proc.memory.rssMB‚ 3218.37109375 proc.cpu.threads‚17 network.recv‚412338% disk./.usageGB‚1160.5920639038086 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 36.2578125 disk.drbd1260.in‚0 disk.dm-0.in‚ 2527.95703125 disk.sda.in‚0 disk.sda.out‚ 36.2578125$ disk.nvme0n1.out‚25681.939453125 disk.sda3.out‚ 36.2578125# disk.nvme0c0n1.in‚ 2527.95703125* proc.memory.percent‚19.643378257751465 cpu‚24.634482282531007 disk.sda2.out‚0' disk./.usagePercent‚77.888307210188! disk.dm-0.out‚25681.939453125! disk.nvme0n1.in‚ 2527.95703125& disk.nvme0c0n1.out‚25681.939453125( proc.memory.availableMB‚ 1500.3828125 network.sent‚891873’*q“­‚(:ˆ µ¦èÔü´ëß gpu.0.gpu‚67.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚278.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iò?Òn®‚(:ã ĦèÔ¿ºïÞ network.recv‚416704$ disk./.usageGB‚1160.752212524414 disk.dm-1.out‚ 36.8671875 disk.drbd1260.out‚ 0.16796875" disk.dm-0.out‚26096.1552734375( proc.memory.availableMB‚ 1500.2734375 cpu‚24.643486418485722# disk.nvme0c0n1.in‚ 2527.95703125' disk.nvme0c0n1.out‚26096.1552734375 disk.sda.out‚ 36.8671875 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 2527.95703125% disk.nvme0n1.out‚26096.1552734375* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17) disk./.usagePercent‚77.89905491849048 disk.sda3.in‚0 disk.sda3.out‚ 36.8671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2527.95703125$ memory_percent‚90.84305763244629# proc.memory.rssMB‚ 3218.37109375 network.sent‚897749À§ü¯‚(:’ ĦèÔ§ã„ß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚238.36' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.672000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û]wæb°‚(:è Ó¦èÔ…ÊïÞ% disk./.usageGB‚1160.5898818969727 disk.dm-0.in‚ 2527.95703125* proc.memory.percent‚19.643378257751465 cpu‚24.653057257441336 proc.cpu.threads‚17) disk./.usagePercent‚77.88816077387841# disk.nvme0c0n1.in‚ 2527.95703125 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚90.84312915802002) proc.memory.availableMB‚ 1500.26171875# proc.memory.rssMB‚ 3218.37109375 network.recv‚420266 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.16796875 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚26468.3193359375å-2! disk.nvme0n1.in‚ 2527.95703125 network.sent‚902933' disk.nvme0c0n1.out‚26468.3193359375 disk.sda.out‚ 37.42578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 37.42578125% disk.nvme0n1.out‚26468.3193359375 disk.sda3.out‚ 37.42578125ƒí¨Ïœ±‚(:‘ Ó¦èÔ¹þ‘à gpu.0.gpu‚38.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚161.65' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.330000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yƒ¤%n²‚(:ã â¦èÔºŽ´Þ) disk./.usagePercent‚77.88806656310582$ disk./.usageGB‚1160.588478088379' disk.nvme0c0n1.out‚26719.2666015625 disk.dm-1.out‚ 37.77734375" disk.dm-0.out‚26719.2666015625% disk.nvme0n1.out‚26719.2666015625 disk.sda3.in‚0 disk.sda3.out‚ 37.77734375 disk.drbd1260.out‚0.171875 disk.sda.in‚0 disk.sda.out‚ 37.77734375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 2527.95703125! disk.nvme0n1.in‚ 2527.95703125* proc.memory.percent‚19.643378257751465 network.recv‚423652 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.84482192993164 proc.cpu.threads‚17 network.sent‚908008# disk.nvme0c0n1.in‚ 2527.95703125 disk.sda1.in‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1499.984375# proc.memory.rssMB‚ 3218.37109375 cpu‚24.661656985929802g‡(-œ³‚(:‘ â¦èÔê—Åß gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×o9o´‚(:ä ñ¦èÔ–ÿÞ) disk./.usagePercent‚77.81683067465072# disk.nvme0c0n1.in‚ 2680.91796875 disk.sda1.out‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚26835.5361328125 disk.sda3.in‚0 disk.sda3.out‚ 38.07421875* proc.memory.percent‚19.643378257751465' disk.nvme0c0n1.out‚26835.5361328125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3218.37109375 cpu‚24.669265287737527 proc.cpu.threads‚17 network.sent‚914306$ disk./.usageGB‚1159.527011871338 disk.sda.in‚0 disk.drbd1260.out‚0.171875 disk.dm-0.in‚ 2680.91796875" disk.dm-0.out‚26835.5361328125! disk.nvme0n1.in‚ 2680.91796875# memory_percent‚90.8433198928833 network.recv‚428555 disk.sda.out‚ 38.07421875 disk.dm-1.out‚ 38.07421875) proc.memory.availableMB‚ 1500.23046875d{xžµ‚(:“ ñ¦èÔ÷цà gpu.0.gpu‚74.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚42.0 gpu.0.powerWatts‚116.595' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.319000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gªäß“¶‚(:ˆ €§èÔôúÃÝ gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚257.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0' xu·‚(:ê €§èÔÔ¡äÞ network.sent‚920161) disk./.usagePercent‚77.82720768404361 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 2680.91796875& disk.nvme0n1.out‚27116.54052734375 disk.sda3.in‚0$ memory_percent‚90.84789752960205( disk.nvme0c0n1.out‚27116.54052734375 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 2680.91796875) proc.memory.availableMB‚ 1499.48046875* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17 network.recv‚432710# disk.nvme0c0n1.in‚ 2680.91796875 disk.sda.out‚ 38.56640625 disk.sda1.out‚0# disk.dm-0.out‚27116.54052734375 disk.sda3.out‚ 38.56640625# proc.memory.rssMB‚ 3218.37109375% disk./.usageGB‚1159.6816368103027 disk.dm-1.out‚ 38.56640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.17578125 disk.sda2.out‚0 cpu‚24.67635819148384$Å‘*m¸‚(:â §èÔþ¢âß cpu‚24.683433928931983 network.sent‚925406( disk.nvme0c0n1.out‚27417.32568359375 disk.dm-1.out‚ 39.140625# proc.memory.rssMB‚ 3218.37109375 proc.cpu.threads‚17 network.recv‚436189 disk.dm-0.in‚ 2680.91796875! disk.nvme0n1.in‚ 2680.91796875 disk.sda3.out‚ 39.140625# disk.nvme0c0n1.in‚ 2680.91796875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚90.84339141845703) disk./.usagePercent‚77.83766277573446% disk./.usageGB‚1159.8374252319336# disk.dm-0.out‚27417.32568359375 disk.sda3.in‚0 disk.sda.out‚ 39.140625 disk.sda2.in‚0& proc.memory.availableMB‚ 1500.21875& disk.nvme0n1.out‚27417.32568359375 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.17578125 disk.sda2.out‚0* proc.memory.percent‚19.643378257751465Y“¹‚(:ˆ §èÔ…†îß gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚259.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïr¿jžº‚(:“ ž§èÔ§±èÞ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.494' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.298800000000014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šF‰Il»‚(:á ž§èÔ’ŽÒà disk.sda.out‚ 39.546875! disk.nvme0n1.in‚ 2680.91796875 disk.sda3.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1499.97265625 proc.cpu.threads‚17 disk.dm-0.in‚ 2680.91796875# disk.nvme0c0n1.in‚ 2680.91796875 disk.sda1.out‚0 disk.dm-1.out‚ 39.546875 disk.sda2.in‚0$ memory_percent‚90.84489345550537# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 disk.sda.in‚0 network.sent‚930325 network.recv‚439739% disk./.usageGB‚1159.8132629394531" disk.dm-0.out‚27684.9775390625 disk.sda3.out‚ 39.546875' disk.nvme0c0n1.out‚27684.9775390625 disk.sda1.in‚0) disk./.usagePercent‚77.83604122401258% disk.nvme0n1.out‚27684.9775390625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1796875 cpu‚24.690906798051344Ó2¼- _wandbruntime‚1299  _runtime‚1299±ŸºAŽ¿ % _timestamp‚1.7884824718435454e+09 _runtime‚1299.386061617 _step‚32 & lr-AdamW/pg1‚0.0007534773431560909 & lr-AdamW/pg2‚0.0007534773431560909 trainer/global_step‚449‰í'™É¾Ã & lr-AdamW/pg1‚0.0007534773431560909 & lr-AdamW/pg2‚0.0007534773431560909 trainer/global_step‚449 % _timestamp‚1.7884824718435454e+09 _runtime‚1299.386061617 _step‚32 iê‹™2¿- _wandbruntime‚1299  _runtime‚1299î‘Pë'ÀåO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137193562112912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10977449 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10952659 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137189969853928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 trainer/global_step‚449 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143001114913792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141651408261632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9123111 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2480578 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1349681224192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1347839936384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142997521317888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135845696748560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28006448687104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95556608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1731793392640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2480575 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10977449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1346113056256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3825606 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1349681224192 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6306184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6305341 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142997521317888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141651433689600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1731888949248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1349681224192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1347839936384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142997521317888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135845696748560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28006448687104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95556608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1731793392640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2480575 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10977449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1346113056256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3825606 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1349681224192 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6306184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6305341 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142997521317888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141651433689600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1731888949248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141965999278080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9143355 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2486062 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6320111 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3834049 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9143361 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29803645298176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11001767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 _step‚34 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141965999278080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20120306 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚449 _runtime‚1302.680267457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1350830449206 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20120306 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10976951 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9143355 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1349107642880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1735748375552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137498223622934 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137494631363742x­,oT T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143318700530176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11001767 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6319278 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20145128 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3833219 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1347263617982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚97640448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28067994563072 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143318700530176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28068023661056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9143361 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141966024706048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2486059 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141966024706048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29803772036608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126738432 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6319278 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20145128 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3833219 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1347263617982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚97640448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28067994563072 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143318700530176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28068023661056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9143361 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141966024706048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2486059 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141966024706048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29803772036608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126738432 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141965999278080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9143355 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2486062 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6320111 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3834049 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9143361 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29803645298176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11001767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 _step‚34 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141965999278080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20120306 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚449 _runtime‚1302.680267457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1350830449206 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20120306 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10976951 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 985841664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9143355 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1349107642880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1735748375552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0"G»ÐX2Å- _wandbruntime‚1302  _runtime‚1302jVERRÆÌ epoch‚0 /loss/FRt3/assignment_loss‚6.056520346937509 2loss/SRqqt1/assignment_loss‚2.8999864878979595 1loss/SRqqt3/assignment_loss‚2.684435708956285 0loss/FBt2/assignment_loss‚0.2986191645555958 0loss/FBt4/assignment_loss‚0.3690112489793036 /loss/FRt4/detection_loss‚1.7665105060482702 $loss/total_loss‚65.4815848345018 /loss/FRt4/assignment_loss‚3.484491401107599 0loss/FBt3/assignment_loss‚0.5196299059894045 .loss/FRt3/detection_loss‚2.847936359458956 /loss/FRt1/assignment_loss‚6.370859506395128 1loss/SRqqt2/assignment_loss‚2.773678421974182 0loss/FBt1/assignment_loss‚0.4615131465327741 % _timestamp‚1.7884824751429746e+09 _runtime‚1302.686452432 /loss/FRt2/assignment_loss‚4.136698476397074 -loss/FRt1/detection_loss‚3.26715374522739 /loss/FBt1/detection_loss‚3.6085986231382075 /loss/FBt2/detection_loss‚1.2424433977712857 trainer/global_step‚449 _step‚35 1loss/SRqqt4/detection_loss‚0.9189112983802532 .loss/FBt4/detection_loss‚4.484596810407108 2loss/SRqqt4/assignment_loss‚2.0138519868082927 /loss/FRt2/detection_loss‚2.1227580569684505 1loss/SRqqt1/detection_loss‚2.8391354219479994 1loss/SRqqt2/detection_loss‚3.6516512712607017 1loss/SRqqt3/detection_loss‚2.9964221504601567 /loss/FBt3/detection_loss‚3.6661713909019125%Ð3~VÇÐ 1loss/SRqqt4/detection_loss‚0.9189112983802532 .loss/FBt4/detection_loss‚4.484596810407108 2loss/SRqqt4/assignment_loss‚2.0138519868082927 /loss/FRt2/detection_loss‚2.1227580569684505 1loss/SRqqt1/detection_loss‚2.8391354219479994 1loss/SRqqt2/detection_loss‚3.6516512712607017 1loss/SRqqt3/detection_loss‚2.9964221504601567 /loss/FBt3/detection_loss‚3.6661713909019125 epoch‚0 /loss/FRt3/assignment_loss‚6.056520346937509 2loss/SRqqt1/assignment_loss‚2.8999864878979595 1loss/SRqqt3/assignment_loss‚2.684435708956285 0loss/FBt2/assignment_loss‚0.2986191645555958 0loss/FBt4/assignment_loss‚0.3690112489793036 /loss/FRt4/detection_loss‚1.7665105060482702 $loss/total_loss‚65.4815848345018 /loss/FRt4/assignment_loss‚3.484491401107599 0loss/FBt3/assignment_loss‚0.5196299059894045 .loss/FRt3/detection_loss‚2.847936359458956 /loss/FRt1/assignment_loss‚6.370859506395128 1loss/SRqqt2/assignment_loss‚2.773678421974182 0loss/FBt1/assignment_loss‚0.4615131465327741 % _timestamp‚1.7884824751429746e+09 _runtime‚1302.686452432 /loss/FRt2/assignment_loss‚4.136698476397074 -loss/FRt1/detection_loss‚3.26715374522739 /loss/FBt1/detection_loss‚3.6085986231382075 /loss/FBt2/detection_loss‚1.2424433977712857 trainer/global_step‚449 _step‚35#`ÈWoÈ‚(:ä ­§èÔâëšà disk.drbd1260.out‚ 0.1796875% disk.nvme0n1.out‚27901.6103515625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 2760.7578125 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚936502 disk.sda2.out‚0 disk.dm-0.in‚ 2760.7578125 disk.sda3.out‚ 39.87109375 disk.sda.out‚ 39.87109375 disk.sda1.out‚0 cpu‚24.699715097754584% disk./.usageGB‚1159.8366966247559 disk.dm-1.in‚0" disk.dm-0.out‚27901.6103515625 disk.nvme0n1.in‚ 2760.7578125) proc.memory.availableMB‚ 1499.45703125# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17 network.recv‚444540) disk./.usagePercent‚77.83761387829541 disk.sda2.in‚0' disk.nvme0c0n1.out‚27901.6103515625$ memory_percent‚90.84804058074951 disk.dm-1.out‚ 39.87109375 disk.drbd1260.in‚0Q¦žÉ‚(:“ ­§èÔ½íÚà gpu.0.gpu‚95.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.081' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.416199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÍìrÊ‚(:ç ¼§èÔ²Ë´ß disk.sda3.out‚ 40.23828125$ memory_percent‚90.84808826446533) proc.memory.availableMB‚ 1499.44921875% disk./.usageGB‚1159.5700607299805 disk.dm-0.in‚ 2800.0546875 cpu‚24.70727221788356 proc.cpu.threads‚17 network.recv‚450130( disk.nvme0c0n1.out‚28074.13232421875 disk.sda.out‚ 40.23828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.18359375# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 network.sent‚986278) disk./.usagePercent‚77.81971971967462" disk.nvme0c0n1.in‚ 2800.0546875 disk.dm-1.out‚ 40.23828125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 2800.0546875 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚28074.13232421875& disk.nvme0n1.out‚28074.13232421875­½žË‚(:“ ¼§èÔóšà gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚253.618' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.723600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>8Ǟ̂(:“ ˧èÔÉõœÞ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚231.368' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.273599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëâ#(eÍ‚(:Ú ˧èÔ¾¸—ß cpu‚24.71358508341525 proc.cpu.threads‚17% disk./.usageGB‚1159.5761375427246& disk.nvme0n1.out‚28226.70263671875 disk.sda3.out‚40.6875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.18359375$ memory_percent‚90.84525108337402( proc.memory.availableMB‚ 1499.9140625 network.sent‚990458 network.recv‚452937 disk.nvme0n1.in‚ 2800.0546875 disk.sda3.in‚0( disk.nvme0c0n1.out‚28226.70263671875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2800.0546875# disk.dm-0.out‚28226.70263671875 disk.sda.out‚40.6875 disk.sda1.in‚0# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465) disk./.usagePercent‚77.82012753967666" disk.nvme0c0n1.in‚ 2800.0546875 disk.dm-1.out‚40.6875 disk.drbd1260.in‚0_†@g΂(:Ü Ú§èÔ¤ªÊÞ( disk./.usagePercent‚77.8284098954244 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚90.84374904632568$ disk./.usageGB‚1159.699550628662" disk.nvme0c0n1.in‚ 2800.0546875 disk.sda1.out‚0* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17 disk.dm-1.out‚ 41.0234375 network.recv‚456474' disk.nvme0c0n1.out‚28355.5263671875 disk.sda.out‚ 41.0234375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2800.0546875" disk.dm-0.out‚28355.5263671875 disk.drbd1260.out‚0.1875 disk.nvme0n1.in‚ 2800.0546875% disk.nvme0n1.out‚28355.5263671875 disk.sda3.out‚ 41.0234375) proc.memory.availableMB‚ 1500.16015625# proc.memory.rssMB‚ 3218.37109375 cpu‚24.719602236520608 disk.drbd1260.in‚0 network.sent‚995597N.(ržÏ‚(:“ Ú§èÔÈÎòÞ gpu.0.gpu‚71.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚251.385' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.276999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Ê%nЂ(:ã é§èÔ´òŽá network.recv‚461393" disk.dm-0.out‚28764.0029296875" disk.nvme0c0n1.in‚ 2800.0546875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 41.3515625 disk.drbd1260.in‚0 disk.dm-0.in‚ 2800.0546875 disk.nvme0n1.in‚ 2800.0546875% disk.nvme0n1.out‚28764.0029296875 disk.sda3.in‚0 disk.sda3.out‚ 41.3515625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚77.84570934854315% disk./.usageGB‚1159.9573249816895 disk.sda.out‚ 41.3515625( proc.memory.availableMB‚ 1500.1484375 cpu‚24.726250826478882' disk.nvme0c0n1.out‚28764.0029296875 disk.sda.in‚0 disk.drbd1260.out‚0.1875$ memory_percent‚90.84382057189941# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17 network.sent‚ 1.001819e+06ØœMÑ‚(:’ é§èÔ’çÔã gpu.0.gpu‚81.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚311.518' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚62.30359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’µzÒ‚(:ï ø§èÔ÷Úä disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2936.66015625& disk.nvme0n1.out‚29220.27587890625 disk.sda3.in‚0$ memory_percent‚90.84386825561523) disk./.usagePercent‚77.85409410730459 disk.sda.out‚ 41.82421875 disk.sda1.in‚0 disk.dm-1.out‚ 41.82421875# disk.dm-0.out‚29220.27587890625' proc.memory.availableMB‚ 1500.140625# proc.memory.rssMB‚ 3218.37109375% disk./.usageGB‚1160.0822639465332( disk.nvme0c0n1.out‚29220.27587890625 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.19140625 disk.sda3.out‚ 41.82421875* proc.memory.percent‚19.643378257751465 cpu‚24.733582594505844 network.sent‚ 1.007665e+06 network.recv‚465384 disk.sda2.in‚0! disk.nvme0n1.in‚ 2936.66015625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 2936.66015625´ˆ¡Ó‚(:’ ø§èÔËî¦è gpu.0.gpu‚90.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚48.0 gpu.0.powerWatts‚331.817' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.36340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,wÔ‚(:ì ‡¨èÔ»Œšß disk.sda3.in‚0( disk.nvme0c0n1.out‚29452.64697265625 disk.sda1.out‚0' proc.memory.availableMB‚ 1500.140625* proc.memory.percent‚19.643378257751465 disk.dm-1.out‚ 42.3515625 disk.sda1.in‚0# proc.memory.rssMB‚ 3218.37109375 cpu‚24.739723804989293 proc.cpu.threads‚17% disk./.usageGB‚1160.2354583740234 disk.sda2.out‚0& disk.nvme0n1.out‚29452.64697265625 disk.sda3.out‚ 42.3515625 disk.sda.in‚0 disk.sda.out‚ 42.3515625$ memory_percent‚90.84386825561523) disk./.usagePercent‚77.86437511387216 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.19140625 disk.sda2.in‚0 disk.dm-0.in‚ 2936.66015625# disk.dm-0.out‚29452.64697265625! disk.nvme0n1.in‚ 2936.66015625# disk.nvme0c0n1.in‚ 2936.66015625 network.sent‚ 1.012862e+06 network.recv‚469090 disk.drbd1260.in‚0k„'¨“Õ‚(:ˆ ‡¨èÔž³Æà gpu.0.gpu‚56.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚252.747' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.5494 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¥ý~nÖ‚(:ã –¨èÔÅŽ­ß' disk.nvme0c0n1.out‚29681.6162109375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 2936.66015625! disk.nvme0n1.in‚ 2936.66015625 disk.sda3.in‚0$ memory_percent‚90.84694385528564# proc.memory.rssMB‚ 3218.37109375) disk./.usagePercent‚77.88188551719912 disk.sda.out‚42.71875) proc.memory.availableMB‚ 1499.63671875* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17 network.sent‚ 1.017882e+06# disk.nvme0c0n1.in‚ 2936.66015625 disk.drbd1260.out‚ 0.1953125 disk.sda2.in‚0" disk.dm-0.out‚29681.6162109375 disk.sda3.out‚42.71875 cpu‚24.74638616254364 network.recv‚472547% disk./.usageGB‚1160.4963760375977 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚42.71875 disk.sda2.out‚0% disk.nvme0n1.out‚29681.6162109375ŠQw“ׂ(:ˆ –¨èÔðÅà gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚225.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.1532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¶ô؞؂(:“ ¥¨èÔݶÞ gpu.0.gpu‚46.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚42.0 gpu.0.powerWatts‚177.322' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.464400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê$¿wÙ‚(:ì ¥¨èÔÜŽïß' disk.nvme0c0n1.out‚29842.9755859375 disk.sda.in‚0 disk.sda.out‚ 42.94921875* proc.memory.percent‚19.643378257751465 network.recv‚477066 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 2936.66015625 cpu‚24.753736826480928 proc.cpu.threads‚17 network.sent‚ 1.024136e+06 disk.dm-0.in‚ 2936.66015625" disk.dm-0.out‚29842.9755859375! disk.nvme0n1.in‚ 2936.66015625$ memory_percent‚90.84689617156982) proc.memory.availableMB‚ 1499.64453125# proc.memory.rssMB‚ 3218.37109375) disk./.usagePercent‚77.88607072836888 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.1953125% disk.nvme0n1.out‚29842.9755859375 disk.sda3.in‚0 disk.sda3.out‚ 42.94921875$ disk./.usageGB‚1160.558738708496 disk.dm-1.out‚ 42.94921875 disk.sda2.in‚0Ú &2Ú- _wandbruntime‚1434  _runtime‚1434¬øeÆÅÛ¿ _step‚36 & lr-AdamW/pg1‚0.0008373835060910675 & lr-AdamW/pg2‚0.0008373835060910675 trainer/global_step‚499 % _timestamp‚1.7884826073546302e+09 _runtime‚1434.896806371¦1™BÉÜà & lr-AdamW/pg2‚0.0008373835060910675 trainer/global_step‚499 % _timestamp‚1.7884826073546302e+09 _runtime‚1434.896806371 _step‚36 & lr-AdamW/pg1‚0.0008373835060910675$!Ëm<2Ý- _wandbruntime‚1434  _runtime‚1434 p Ò½ÞèO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚814 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157380984512000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157380984512000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4251091 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10135605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1499438383616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10135611 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12195401 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7005055 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31083742485504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12170611 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22306216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_qdìg1split_bytes.small_pool.peak‚ 761954816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018241024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7005869 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1493825908076 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1495870215680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33008406532608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158876829299712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10135605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12195401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152423072516636 trainer/global_step‚499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1499438383616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 122557440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31083771583488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152426664775620 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚814 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157380984512000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157380984512000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4251091 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10135605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1499438383616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10135611 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12195401 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7005055 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31083742485504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12170611 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22306216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761954816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018241024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7005869 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1493825908076 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1495870215680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3593595904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33008406532608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158876829299712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10135605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12195401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152423072516636 trainer/global_step‚499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1499438383616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 122557440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6%A02à- _wandbruntime‚1437  _runtime‚1437¢+Ý÷&áñM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10155861 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151230917605760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10155855 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12194903 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159198008512000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159194414902784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1502432983552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2760259 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1498864802304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12194903 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚817 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22375580 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33073945359360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152731326285642 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22350758 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4258607 _runtime‚1437.356681895 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 122544128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10155855 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152727734026450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157695550100480 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1502432983552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7018866 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚93446144 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157695575528448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 # _timestamp‚1.78848260981397e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 trainer/global_step‚499 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7018866 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚93446144 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157695575528448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 # _timestamp‚1.78848260981397e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 trainer/global_step‚499 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22375580 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33073945359360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152731326285642 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22350758 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4258607 _runtime‚1437.356681895 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 122544128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10155855 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152727734026450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157695550100480 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224&a6©L2ã- _wandbruntime‚1437  _runtime‚1437ÈFï«SäÍ -loss/FBt4/detection_loss‚3.42526838183403 &loss/total_loss‚61.078925625808864 2loss/SRqqt3/assignment_loss‚2.6825090487798056 1loss/SRqqt4/assignment_loss‚1.787495537378797 1loss/SRqqt1/detection_loss‚3.5485214069485664 1loss/SRqqt3/detection_loss‚2.9763109644254047 0loss/FBt4/assignment_loss‚0.7469749071381309 /loss/FRt1/detection_loss‚3.3334288441616557 /loss/FBt1/detection_loss‚2.9488055790370367 /loss/FRt1/assignment_loss‚5.520787840304167 1loss/SRqqt2/assignment_loss‚2.789460316300392 0loss/FBt1/assignment_loss‚0.4701384619940377 epoch‚0 0loss/FRt2/assignment_loss‚3.7204279942171916 1loss/SRqqt1/assignment_loss‚2.635648738592863 1loss/SRqqt4/detection_loss‚0.8674568253089533 _runtime‚1437.357871608 % _timestamp‚1.7884826098164907e+09 /loss/FRt3/assignment_loss‚5.686809447274279 0loss/FBt2/assignment_loss‚0.3181502690159062 .loss/FRt2/detection_loss‚2.196214202715426 /loss/FRt3/detection_loss‚2.6903773136992952 0loss/SRqqt2/detection_loss‚3.552958734333515 /loss/FBt2/detection_loss‚0.9948724347380443 /loss/FBt3/detection_loss‚3.1587399661540987 /loss/FRt4/assignment_loss‚2.749093423980151 0loss/FBt3/assignment_loss‚0.4251471608875416 _step‚39 trainer/global_step‚499 .loss/FRt4/detection_loss‚1.853327826589577Në¾0WåÑ 1loss/SRqqt4/assignment_loss‚1.787495537378797 1loss/SRqqt1/detection_loss‚3.5485214069485664 1loss/SRqqt3/detection_loss‚2.9763109644254047 0loss/FBt4/assignment_loss‚0.7469749071381309 /loss/FRt1/detection_loss‚3.3334288441616557 /loss/FBt1/detection_loss‚2.9488055790370367 /loss/FRt1/assignment_loss‚5.520787840304167 1loss/SRqqt2/assignment_loss‚2.789460316300392 0loss/FBt1/assignment_loss‚0.4701384619940377 epoch‚0 0loss/FRt2/assignment_loss‚3.7204279942171916 1loss/SRqqt1/assignment_loss‚2.635648738592863 1loss/SRqqt4/detection_loss‚0.8674568253089533 _runtime‚1437.357871608 % _timestamp‚1.7884826098164907e+09 /loss/FRt3/assignment_loss‚5.686809447274279 0loss/FBt2/assignment_loss‚0.3181502690159062 .loss/FRt2/detection_loss‚2.196214202715426 /loss/FRt3/detection_loss‚2.6903773136992952 0loss/SRqqt2/detection_loss‚3.552958734333515 /loss/FBt2/detection_loss‚0.9948724347380443 /loss/FBt3/detection_loss‚3.1587399661540987 /loss/FRt4/assignment_loss‚2.749093423980151 0loss/FBt3/assignment_loss‚0.4251471608875416 _step‚39 trainer/global_step‚499 .loss/FRt4/detection_loss‚1.853327826589577 -loss/FBt4/detection_loss‚3.42526838183403 &loss/total_loss‚61.078925625808864 2loss/SRqqt3/assignment_loss‚2.6825090487798056'3Q—’æ‚(:‡ ´¨èÔ–…Ëâ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚246.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.271 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ûÒ¯tç‚(:é ´¨èÔܱ»ã disk.sda1.out‚0 disk.dm-1.out‚ 43.4296875" disk.dm-0.out‚30020.6962890625 disk.sda3.out‚ 43.4296875( proc.memory.availableMB‚ 1499.9921875' disk.nvme0c0n1.out‚30020.6962890625 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3087.29296875! disk.nvme0n1.in‚ 3087.29296875% disk.nvme0n1.out‚30020.6962890625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.19921875 network.sent‚ 1.029976e+06 network.recv‚481520 disk.sda.out‚ 43.4296875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 cpu‚24.760012286310857) disk./.usagePercent‚77.89026131569686# disk.nvme0c0n1.in‚ 3087.29296875$ memory_percent‚90.84477424621582# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17$ disk./.usageGB‚1160.621181488037›IDuè‚(:ê èèÔ…ÞÓß disk.sda.out‚ 43.92578125 disk.dm-1.in‚0% disk.nvme0n1.out‚30462.5908203125 disk.sda3.in‚0 network.sent‚ 1.079082e+06 disk.sda1.out‚0" disk.dm-0.out‚30462.5908203125! disk.nvme0n1.in‚ 3087.29296875& proc.memory.availableMB‚ 1500.03125 cpu‚24.76657607907891% disk./.usageGB‚1161.3735961914062# disk.nvme0c0n1.in‚ 3087.29296875 disk.drbd1260.out‚ 0.19921875 disk.sda3.out‚ 43.92578125$ memory_percent‚90.84453582763672# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 43.92578125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3087.29296875 proc.cpu.threads‚17 network.recv‚486586) disk./.usagePercent‚77.94075649775795' disk.nvme0c0n1.out‚30462.5908203125%ï_‘žé‚(:“ èèÔ³û†à gpu.0.gpu‚76.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚219.854' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.970800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ÂýBwê‚(:ì Ò¨èÔèŒùß$ memory_percent‚90.84451198577881 cpu‚24.773037732511153 proc.cpu.threads‚17& disk.nvme0n1.out‚30963.60205078125# disk.nvme0c0n1.in‚ 3087.29296875( disk.nvme0c0n1.out‚30963.60205078125 disk.sda1.out‚0 network.sent‚ 1.084082e+06 network.recv‚490330# disk.dm-0.out‚30963.60205078125 disk.sda3.in‚0 disk.sda3.out‚ 44.1796875 disk.sda.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1500.03515625% disk./.usageGB‚1161.6210632324219 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3087.29296875! disk.nvme0n1.in‚ 3087.29296875 disk.drbd1260.out‚0.203125# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465) disk./.usagePercent‚77.95736421851919 disk.sda.out‚ 44.1796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 44.1796875É6¤B“ë‚(:ˆ Ò¨èÔÍâ¶à gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚249.282' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹yåÄtì‚(:é á¨èÔ¬ñÞ disk.dm-1.in‚0) disk./.usagePercent‚77.96578763441829# disk.dm-0.out‚31309.13720703125! disk.nvme0n1.in‚ 3087.29296875 disk.sda3.out‚ 44.5234375# disk.nvme0c0n1.in‚ 3087.29296875 disk.sda1.out‚0 disk.dm-1.out‚ 44.5234375 disk.drbd1260.out‚0.203125% disk./.usageGB‚1161.7465782165527( disk.nvme0c0n1.out‚31309.13720703125$ memory_percent‚90.84453582763672# proc.memory.rssMB‚ 3218.37109375 cpu‚24.778940289144657 proc.cpu.threads‚17 network.sent‚ 1.090348e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 1500.03125* proc.memory.percent‚19.643378257751465 network.recv‚495144 disk.sda2.out‚0 disk.dm-0.in‚ 3087.29296875& disk.nvme0n1.out‚31309.13720703125 disk.sda3.in‚0 disk.sda.out‚ 44.5234375 disk.sda1.in‚0¨E’í‚(:‡ á¨èÔð™¼à gpu.0.gpu‚32.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚43.0 gpu.0.powerWatts‚248.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯.8‚“î‚(:ˆ ð¨èÔ€ãþÞ gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚247.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɵc†yï‚(:î ð¨èÔÉç ß disk.dm-1.in‚0 disk.dm-1.out‚ 45.04296875 disk.drbd1260.in‚0& disk.nvme0n1.out‚31583.98681640625 disk.sda3.out‚ 45.04296875( disk.nvme0c0n1.out‚31583.98681640625 disk.sda.in‚0# disk.dm-0.out‚31583.98681640625 network.recv‚499315% disk./.usageGB‚1161.9970817565918 disk.sda1.out‚0 disk.drbd1260.out‚ 0.20703125 disk.sda2.out‚0 disk.dm-0.in‚ 3248.95703125! disk.nvme0n1.in‚ 3248.95703125 network.sent‚ 1.098097e+06 disk.sda1.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 3248.95703125 disk.sda.out‚ 45.04296875$ memory_percent‚90.84692001342773' proc.memory.availableMB‚ 1499.640625# proc.memory.rssMB‚ 3218.37109375) disk./.usagePercent‚77.98259913717678 disk.sda2.in‚0* proc.memory.percent‚19.643378257751465 cpu‚24.78452786504087 proc.cpu.threads‚17‰ÿÓøvð‚(:ë ÿ¨èÔ¡¡åß$ memory_percent‚90.84539413452148 cpu‚24.790782936803566 disk.dm-0.in‚ 3248.95703125! disk.nvme0n1.in‚ 3248.95703125& disk.nvme0n1.out‚31978.74072265625( disk.nvme0c0n1.out‚31978.74072265625# proc.memory.rssMB‚ 3218.37109375* proc.memory.percent‚19.643378257751465 proc.cpu.threads‚17( disk./.usagePercent‚78.0162216306787 disk.dm-1.in‚0 disk.dm-1.out‚ 45.7265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.20703125# disk.nvme0c0n1.in‚ 3248.95703125' proc.memory.availableMB‚ 1499.890625 network.sent‚ 1.101325e+06 network.recv‚502796% disk./.usageGB‚1162.4980812072754 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.dm-0.out‚31978.74072265625 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 45.7265625 disk.sda.in‚0 disk.sda.out‚ 45.7265625ÔÑ»“ñ‚(:ˆ ÿ¨èÔœ·ýß gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚232.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷8ƒ¤sò‚(:è Ž©èÔ‹úšÞ proc.cpu.threads‚17 network.sent‚ 1.106287e+06" disk.dm-0.out‚32373.5107421875 disk.dm-1.out‚ 46.1484375( proc.memory.availableMB‚ 1499.6484375 cpu‚24.796526466737994( disk./.usagePercent‚78.0152997475479 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚32373.5107421875 disk.sda3.in‚0 disk.sda.out‚ 46.1484375 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚506102 disk.dm-0.in‚ 3248.95703125 disk.sda3.out‚ 46.1484375' disk.nvme0c0n1.out‚32373.5107421875 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2109375* proc.memory.percent‚19.643378257751465% disk./.usageGB‚1162.4843444824219! disk.nvme0n1.in‚ 3248.95703125# disk.nvme0c0n1.in‚ 3248.95703125 disk.sda1.in‚0$ memory_percent‚90.84687232971191# proc.memory.rssMB‚ 3218.37109375ïÀOžó‚(:“ Ž©èÔûÍçà gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚115.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.131200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05pjô‚(:ß ©èÔ×éýß disk.sda1.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 1.112639e+06 network.recv‚510768 disk.sda3.out‚46.6875# disk.nvme0c0n1.in‚ 3248.95703125 disk.dm-1.in‚0 disk.dm-1.out‚46.6875 disk.drbd1260.out‚ 0.2109375' proc.memory.availableMB‚ 1499.640625% disk./.usageGB‚1162.4960632324219 disk.dm-0.in‚ 3248.95703125% disk.nvme0n1.out‚32717.9677734375 disk.sda2.out‚0# proc.memory.rssMB‚ 3218.37109375 cpu‚24.802366046219188" disk.dm-0.out‚32717.9677734375' disk.nvme0c0n1.out‚32717.9677734375 disk.sda.out‚46.6875 disk.sda2.in‚0$ memory_percent‚90.84692001342773* proc.memory.percent‚19.643378257751465) disk./.usagePercent‚78.01608620269307! disk.nvme0n1.in‚ 3248.95703125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0[«¨m“õ‚(:ˆ ©èÔ¿¢ùá gpu.0.gpu‚97.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚231.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.3852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÆîöqö‚(:æ ¬©èԀБà) disk./.usagePercent‚78.04977705020664% disk./.usageGB‚1162.9980812072754 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 47.07421875 disk.sda2.in‚0# disk.dm-0.out‚33172.88427734375 network.recv‚515187 disk.nvme0c0n1.in‚ 3391.15625 disk.sda.out‚ 47.07421875# memory_percent‚90.8484697341919) proc.memory.availableMB‚ 1499.38671875# proc.memory.rssMB‚ 3218.37109375 proc.cpu.threads‚17 network.sent‚ 1.118644e+06& disk.nvme0n1.out‚33172.88427734375 disk.sda3.out‚ 47.07421875( disk.nvme0c0n1.out‚33172.88427734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.21484375 disk.sda2.out‚0 cpu‚24.80799484695437 disk.nvme0n1.in‚ 3391.15625 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3391.15625* proc.memory.percent‚19.643378257751465{ˆŠ’÷‚(:‡ ¬©èÔݯ¯à gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚235.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰o…’ø‚(:‡ »©èÔÐí‰à gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚244.435' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.887 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’nÊoù‚(:ä »©èÔƒõèá disk.sda2.out‚0 disk.dm-0.in‚ 3391.15625# disk.dm-0.out‚33423.35302734375# memory_percent‚90.8393144607544) proc.memory.availableMB‚ 1500.88671875 cpu‚24.813624857033624 proc.cpu.threads‚17 disk.dm-1.out‚ 47.6484375 disk.drbd1260.out‚ 0.21484375 network.recv‚518580) disk./.usagePercent‚78.04471142912662 disk.sda1.in‚0 disk.nvme0n1.in‚ 3391.15625& disk.nvme0n1.out‚33423.35302734375* proc.memory.percent‚19.643378257751465( disk.nvme0c0n1.out‚33423.35302734375 disk.drbd1260.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3218.37109375 disk.dm-1.in‚0 network.sent‚ 1.123725e+06% disk./.usageGB‚1162.9225997924805 disk.nvme0c0n1.in‚ 3391.15625 disk.sda.out‚ 47.6484375 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 47.6484375 disk.sda.in‚0øÛÛ2ú- _wandbruntime‚1583  _runtime‚1583H?îNÅû¿ _step‚40 & lr-AdamW/pg1‚0.0009212896690260444 & lr-AdamW/pg2‚0.0009212896690260444 trainer/global_step‚549 % _timestamp‚1.7884827554641411e+09 _runtime‚1583.006612135”èo1Éüà trainer/global_step‚549 % _timestamp‚1.7884827554641411e+09 _runtime‚1583.006612135 _step‚40 & lr-AdamW/pg1‚0.0009212896690260444 & lr-AdamW/pg2‚0.0009212896690260444(±2ý- _wandbruntime‚1583  _runtime‚1583„N™Uï'þéO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3028975 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4677564 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167659767438328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3028978 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13413353 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13388563 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7706542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24536668 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174756137281536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1645627375104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚41 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13413353 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚91362304 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1643378710784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 % _timestamp‚1.7884827554676318e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34161065381888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 120460288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13388563 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 _runtime‚1583.010532334 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7705719 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173110535334400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4676744 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36278693302784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11148105 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2117536558592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308Í6‚P·ÿíO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166012796468560 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1645627375104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173110509906432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11148105 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚823 QIDeviceStatsMonitor.on_train_batch_st1ƒq<%art/inactive_split.small_pool.current‚820 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34161036283904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2117627920896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2117536558592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3028975 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4677564 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167659767438328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537323349 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3028978 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13413353 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13388563 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7706542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24536668 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174756137281536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1645627375104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚41 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13413353 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚91362304 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1643378710784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 % _timestamp‚1.7884827554676318e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34161065381888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24790 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 120460288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13388563 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 _runtime‚1583.010532334 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7705719 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173110535334400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4676744 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36278693302784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11148105 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 _runtime‚1585.972108557 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11168355 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34222582159872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 # _timestamp‚1.78848275842858e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175077316493824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11168361 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7719777 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167964428948350 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1646369223398 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1652190142976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166314492893728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2121499179520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167960836689158 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1652190142976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166314467465760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 trainer/global_step‚549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1648621961728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11168355 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚790 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34222611257856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13437671 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13412855 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13412855 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚91348992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173425126350848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1648621961728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4685318 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24606032 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7720567 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24581210 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2121407830528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173425100922880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13437671 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175077316493824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24581210 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚787 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3034462 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11168361 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173425100922880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚42ŒÈàû&‚õM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1649936054622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13412855 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13412855 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚91348992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173425126350848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1648621961728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4685318 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568181248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24606032 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7720567 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24581210 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2121407830528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173425100922880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13437671 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175077316493824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24581210 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚787 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3034462 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11168361 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173425100922880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚42 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 _runtime‚1585.972108557 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11168355 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34222582159872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 # _timestamp‚1.78848275842858e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175077316493824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11168361 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7719777 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167964428948350 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1646369223398 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1652190142976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166314492893728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456804352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2121499179520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167960836689158 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455445764 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1652190142976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166314467465760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 trainer/global_step‚549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1648621961728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11168355 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚790 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34222611257856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13437671 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190635445263360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚93459456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26767120 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188840086156800 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5103370 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2310726753792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14631305 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795384534528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14606515 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761954816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456804352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12160605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12160611 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37238330082304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181096371756528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181096346328560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5104188 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39549179393536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1792931513492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795384534528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1796498344508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188840060728832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1798952702464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37238359180288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190639038859264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14631305 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182889277842052 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 _runtime‚1731.755348255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 985841664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1798952702464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190639038859264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26791916 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12160605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚599 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39549056836096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190635445263360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚93459456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26767120 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188840086156800 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5103370 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2310726753792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30388 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14631305 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24790 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795384534528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14606515 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761954816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456804352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30637 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12160605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455445764 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30637 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12160611 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30388 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37238330082304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592258984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568167936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181096371756528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181096346328560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24796 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5104188 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39549179393536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1792931513492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795384534528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566831016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568167936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1796498344508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188840060728832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1798952702464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37238359180288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚Ý5}Q190639038859264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24796 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14631305 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3593595904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182889277842052 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319-§êSF2 - _wandbruntime‚1733  _runtime‚1733ŒËtù&¡óM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 122544128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37299905056256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚93446144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1799488857330 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 % _timestamp‚1.7884829064432323e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12180861 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1798379121152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8421221 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5112559 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 _runtime‚1733.985484408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26811662 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26811662 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚189154651745280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14655623 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚799 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5111763 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚189154677173248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26836484 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 trainer/global_step‚599 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3308662 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39614579688448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26836484 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183193939351866 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14655623 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37299875958272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 _step‚46 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8420422 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183197531611058 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12180855 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14630807 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14630807 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1795922026106fÖpý&¢÷M ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181398017325760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3308659 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚189154677173248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39614457144320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537323349 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018241024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39614579688448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26836484 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592259192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183193939351866 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30637 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14655623 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018241024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37299875958272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 _step‚46 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3593609216 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8420422 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183197531611058 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456804352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12180855 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14630807 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14630807 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1795922026106 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 122544128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37299905056256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚93446144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1799488857330 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 % _timestamp‚1.7884829064432323e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12180861 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3593609216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1798379121152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8421221 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5112559 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 _runtime‚1733.985484408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26811662 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26811662 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566831224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24822 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚189154651745280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568181248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14655623 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚799 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5111763 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚189154677173248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26836484 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 trainer/global_step‚599 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3308662 Pq•NµbI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÒ×ÎêH¶bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¿§Z=H·bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÙV‚SN¸bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP/$<ÅN¹bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP˜j$HºbC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPWÕ¦J»bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPp%çN¼bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP ªg›J½bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPŽXsaN¾bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÑüÞ8N¿bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPófàfJÀbE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP¿HÁbC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP,ŠéeJÂbE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPa8&NÃbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¡`-uJÄbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¶öÖšHÅbC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP–%ØNÆbI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP²ª<JÇbE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÏÜëNÈbI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP^g/«JÉbE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPMT JÊbE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPIÐÇoJËbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP³çì€JÌbE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPC°[2NÍbI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPKœbNÎbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPø¶GÏbE "Purity/0$kC;FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPn^±mHÐbC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP]D³eNÑbI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÉ=u¯HÒbC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP€bAÃJÓbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPeÃòJÔbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP‡Ýf@Õb; particle/accuracy_2_of_4"trainer/global_step2:JP¾ì¡HÖbC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPža·/J×bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPþ¯RHØbC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÓeL.JÙbE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPƒ?2JÚbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP˜ÄµiJÛbE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPpÒEJÜbE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP„¹”EHÝbC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPñ7PNÞbI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPþþíiNßbI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPRw¤JàbE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP˜÷AîJábE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP†ÆTJâbE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÄÿÛNãbI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPæ_µ'HäbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPïP~0NåbI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP8oÝ£JæbE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPžã]Ÿ;çb6 jet/accuracy_6_of_6"trainer/global_step2:JP„Y¼HèbC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP‘Ç‘?HébC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPG…öøJêbE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPñàöJëbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPWTkHìbC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPç;¢uHíbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPð4ÍHîbC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP:¸øJïbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP u,NðbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPº” JñbE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPäßÆJòbE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP GTNóbI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP”S‰ØJôbE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP‰j>¥JõbE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÀeINöbI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPõ}.`J÷bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¶wÓåHøbC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP&7´ŽNùbI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPŠCx JúbE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPç·Ì NûbI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPþ˦JübE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPŒ`ù,NýbI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP'9à NþbI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¼`fNÿbI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÈê^`H€bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPo¶!jJbE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPý>ˆ H‚bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP%e.HƒbC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP%Ö ðN„bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP$Rù“N…bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPä@†b; particle/accuracy_2_of_2"trainer/global_step2:JP (ÒÚH‡bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPç[JJˆbE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPZ½9|N‰bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPp/î¤;Šb6 jet/accuracy_1_of_4"trainer/global_step2:JP¦qßãJ‹bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPx\r.JŒbE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPJF!½JbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP-Í›ðNŽbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPï[ÖöHbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP=õÐJbE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPœÊËŸJ‘bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÁXÁáN’bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP²ˆ5¨N“bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP*CH”bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPï!;•b6 jet/accuracy_4_of_6"trainer/global_step2:JP‚ÛÝH–bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPŠüCN—bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÖ²4fN˜bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPô‘ËH™bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPô7Á¼JšbE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ë®ï@›b; particle/accuracy_4_of_7"trainer/global_step2:JP êHœbC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP’Ë=JbE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPŒ×—ÀHžbC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÜÐÍaHŸbC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP:Æ ÉJ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPý2°ŽJ¡bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP3ZPJ¢bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÁöN£bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP°²î`N¤bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPnÿáN¥bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPá¢c'N¦bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÀÅðºN§bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÈ+.®N¨bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÐ^n@©b; particle/accuracy_4_of_5"trainer/global_step2:JPÚ þoJªbE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP~IÓÙJ«bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPª²|N¬bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPX=5J­bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPãùëÏN®bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¾¹äuJ¯bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPhÀ»¡H°bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP#ÄUN±bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP³7|XH²bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP7í|!N³bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPñ›‚J´bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPâ•üJµbE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP›là“H¶bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¯&f7N·bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPf÷µ]H¸bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPâGRcJ¹bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP ŽQJºbE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP!·­‚J»bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP~àH‡@¼b; particle/accuracy_3_of_6"trainer/global_step2:JPð§¡ÊJ½bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP°ìYLJ¾bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPäLJ¿bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP‹QNÀbI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP#i JÁbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPEâÿ„NÂbI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP3î[FNÃbI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP9–`NÄbI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP²ßÊéHÅbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÝÊùvJÆbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP,¬ÛHÇbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPØŽdÉJÈbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP8ÍÄ;Éb6 jet/accuracy_3_of_5"trainer/global_step2:JPÿKž`;Êb6 jet/accuracy_4_of_7"trainer/global_step2:JPBáýKJËbE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPƒuD)HÌbC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP°ÝYJÍbE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPnöžNÎbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPe>*XNÏbI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP°’]²NÐbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPX*×NÑbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP‘²ÿ`HÒbC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP]4nÄ@Ób; particle/accuracy_2_of_3"trainer/global_step2:JPNÃÑzJÔbE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP ‹qNÕbI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPH%pNÖbI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÓJ×bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÍ– HØbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÖ‡.àJÙbE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP:ÉHÚbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPj“·HÛbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP3NÜbI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPkNl±NÝbI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPY =üHÞbC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPïÏÓóHßbC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPúÀHàbC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP¯;8 HábC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPgZNâbI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPƒ½Ó•HãbC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPXŒøNäbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP T ÉJåbE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¸†/NæbI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP(W˜‡ËN¾bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP.U^N¿bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP/Ÿ`ÆNÀbI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPK±ÃNÁbI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP¤“GNÂbI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP>vANÃbI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPò¬Ž²JÄbE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPÑñHµHÅbC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÂ’íJÆbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPâÑô¥NÇbI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPƒœBÖ@Èb; particle/accuracy_2_of_6"trainer/global_step2:JPw“½@Éb; particle/accuracy_5_of_7"trainer/global_step2:JP}EîËJÊbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP2œÝ.JËbE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÅf·ÒNÌbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP±¸]NÍbI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÁ JÎbE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPöÏJÏbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPj´JÐbE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPMEA@Ñb; particle/accuracy_2_of_7"trainer/global_step2:JP.ñöHÒbC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP+R58JÓbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPì¡É'@Ôb; particle/accuracy_7_of_7"trainer/global_step2:JP\á˜JÕbE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP·gFzHÖbC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPwh4GN×bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPSCkNØbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP• MHÙbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÌ}š.JÚbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPþe JÛbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP°Ø8@Üb; particle/accuracy_4_of_6"trainer/global_step2:JP2ШžHÝbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPê8TNÞbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP )¨YJßbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPo™NàbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP#ÅGMHábC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPâyƒHâbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP)œ¢ŸNãbI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP%'†Ñ;äb6 jet/accuracy_1_of_2"trainer/global_step2:JP^2NåbI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP ¥úNæbI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP…>ßk;çb6 jet/accuracy_2_of_7"trainer/global_step2:JPTÊ1HèbC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPhëHéNébI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPB¹„GêbB validation_average_jet_accuracy"trainer/global_step2:JP³sgNëbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÎÕ§•JìbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP+ЪHíbC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPK;TNîbI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP²£¾HïbC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP 7HðbC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP ºŸ‰HñbC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPE+LJòbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPMhsNóbI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP»÷Æû@ôb; particle/accuracy_1_of_2"trainer/global_step2:JPâÔj>HõbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¼™DNöbI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPèOÊ);÷b6 jet/accuracy_4_of_5"trainer/global_step2:JP“lHøbC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP«R_ÔHùbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÓ‹#`JúbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP-Š~JûbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP­ÝhaJübE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP×ýÐNýbI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPƒÍWNþbI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPNWÌHÿbC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP¨Rü~N€bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP&ÖžNbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÜ©¹îH‚bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPã?|¯NƒbI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP—ŸH„bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¸¡wJ…bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP–4‰âN†bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPüD:H‡bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPy¸YñNˆbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPrŠ¯)J‰bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPv$º’NŠbI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPPÀN‹bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÍc.á;Œb6 jet/accuracy_1_of_1"trainer/global_step2:JPjÌ~ªNbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÌýtJŽbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPeÖ’+NbI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP"”ŠRNbI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP!d©;‘b6 jet/accuracy_2_of_2"trainer/global_step2:JP/¤—ùJ’bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPá”/ H“bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPžÄjBJ”bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPEDàN•bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPN×øN–bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPà ?hN—bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÜeÖ;˜b6 jet/accuracy_6_of_7"trainer/global_step2:JPØ*mJ™bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP§â•-HšbC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP¸¼ÇÂH›bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPõˆ¿JœbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP#²USNbI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP)&£‹HžbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP›ì—JŸbE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPLYJ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPè¡ 2N¡bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP¦…ÜöH¢bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÒÎxçN£bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÝH–N¤bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¢-’J¥bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÌ+J¦bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÿêATJ§bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPõ:IN¨bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP«­±…N©bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP+Cˆ¨NªbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPºvH«bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPnDçfH¬bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP!SÂÜJ­bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‰lS°N®bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÿ2N¯bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÀˆØH°bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPAª6H±bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP€AøH²bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP–âJ³bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP1ñ2'N´bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP´ž¥HµbC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPxŠ%\J¶bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPoŒï¾N·bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¿M¼N¸bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP-ô#ìN¹bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPvƒ0´@ºb; particle/accuracy_1_of_7"trainer/global_step2:JPœ|46H»bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP“Ì(J¼bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPm´O J½bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP-ÿ·J¾bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPz[SOH¿bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPP«]HÀbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPMªãHÁbC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPGØñHÂbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPwíäEJÃbE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP@̲HÄbC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPzÙÇINÅbI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÏ.þžJÆbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¼šåÒNÇbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPíèÁôJÈbE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP7T3ªHÉbC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP0ßR¼JÊbE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP7?=JËbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPCá¹NÌbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPO“ÿ÷HÍbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP ³aNÎbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPQ5î®JÏbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP9JÐbE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPî,’(JÑbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPG$Ÿ¤HÒbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPì NÓbI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP8ÃaHÔbC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP×=—ÎJÕbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPräJÖbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¼¿N×bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPYZaUHØbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP ];ôHÙbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÌôÐiNÚbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP1ç—w@Ûb; particle/accuracy_5_of_5"trainer/global_step2:JPNZ6¬HÜbC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP¯’¨vJÝbE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP× …¹NÞbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP’þtªHßbC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPäѯåNàbI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPèÉO·JábE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPb†‘¢HâbC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPýl)JãbE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP×—æ,-äbC Purity/3FRt0SRqqt2FBt/FRt_purity"trai+ʼnÌner/global_step2:JPàENåbI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP…ÍœóNæbI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP§°4JçbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP —JèbE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP³0Ö;éb6 jet/accuracy_2_of_4"trainer/global_step2:JP   HêbC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÝi}NëbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPm õŽNìbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP àJíbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP€ñ£²NîbI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP L;ïb6 jet/accuracy_2_of_3"trainer/global_step2:JPë}.ŒHðbC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÉšÏJñbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP@­/•JòbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPŠ‚€iHóbC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÇ1ÔEHôbC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP±ÿ‘¦JõbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP)ˆNöbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP'N]HN÷bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¤¨²­NøbI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÕªF©NùbI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP-Üa\JúbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPvkHûbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP‚¹ó¡HübC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÉjHýbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¾0 JþbE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPR1²NÿbI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPzå}N€ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPüor–J bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÍï#J‚ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP:q©fNƒ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPëýXZH„ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPû¡UH… bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP”5èN† bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP‘7SLN‡ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP¢õSaJˆ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÑw1;‰ b6 jet/accuracy_3_of_7"trainer/global_step2:JPãfeˆHŠ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPBN‹ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPx˜kïHŒ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPâ¨J bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¯+HŽ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP6—N bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPM†DJ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP$L€ H‘ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPëÏóJ’ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP«êf N“ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPæDû J” bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP­(^Q@• b; particle/accuracy_5_of_6"trainer/global_step2:JPjÑ33N– bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP@1 úJ— bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPUé)$J˜ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP©ÙJ™ bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP•úd Jš bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP†bIÙ@› b; particle/accuracy_3_of_7"trainer/global_step2:JPO@&Nœ bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP=&˜5H bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPeexÒNž bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPŠÓzqNŸ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPѾñÈH  bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP‚ž§i;¡ b6 jet/accuracy_1_of_6"trainer/global_step2:JPV¤™àH¢ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP9\`J£ bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPçÓ¬¶N¤ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÒuJ¥ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP;ºðÀJ¦ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPævzJ§ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPûµŠJ¨ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPõ›YuN© bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP©4„¡Nª bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPPsŽØN« bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPRN[uH¬ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP´ˆÏ>J­ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPNà!N® bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPŽEâN¯ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP ˆN° bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPœK§.H± bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP]óAÔN² bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP/•2qH³ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPÜ|5óHÅ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÀ€œ6JÆ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPPÛJHÇ bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPsãÅäJÈ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPå}ž+HÉ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÙ£à¿JÊ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPJuézJË bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPãÃ+JÌ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP›—IdHÍ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP.Ù‹ÔHÎ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÖûLí@Ï b; particle/accuracy_6_of_6"trainer/global_step2:JP8‘HÐ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPù"e5NÑ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP]wœDJÒ bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP^Xr¦HÓ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPm¦3JÔ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPµŽ&NÕ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPçægmHÖ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPKý{J× bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPÖA`óJØ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÊ& ¬NÙ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÁl¤;Ú b6 jet/accuracy_1_of_3"trainer/global_step2:JPgÛHÛ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¤ÇJÜ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP©ùSrJÝ bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP·z65NÞ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP~ZiKJß bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP 1jJà bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¦=¢¬Há bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPðocNâ bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP¸àññ@ã b; particle/accuracy_1_of_4"trainer/global_step2:JPÊì ÅHä bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP<“Ä Hå bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPnkbàHæ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÑÌÍNç bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPƒ|F{Hè bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPR—Né bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP;óš Jê bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP±Jë bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÁZ¯8Nì bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPCùãjHí bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP,®Ž¯Nî bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPEžî0Nï bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPf¼CìJð bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPëõNJñ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPHüçEJò bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP™9ƒ Hó bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP»+I•Hô bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPz–m@õ b; particle/accuracy_4_of_4"trainer/global_step2:JP6ߊ@ö b; particle/accuracy_3_of_5"trainer/global_step2:JP—å¬H÷ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPÆ•½úJø bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP=ËFJù bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP*“*Hú bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÔìy…Hû bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPTI¨Jü bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP§³§'Jý bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP˜RRHþ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP›þ¦¯@ÿ b; particle/accuracy_1_of_5"trainer/global_step2:JP€mPÉN€ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP^†ÈñJ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPËŠ^Ò@‚ b; particle/accuracy_1_of_6"trainer/global_step2:JPkbHƒ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPð‡¬J„ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP JóÊN… bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPGD×_H† bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÈù u;‡ b6 jet/accuracy_2_of_5"trainer/global_step2:JP½’xJˆ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP½÷H‰ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPV.þÐJŠ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÂLã‘N‹ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPl7ñJŒ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP†<Ò,J bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPŸÚln;Ž b6 jet/accuracy_3_of_4"trainer/global_step2:JPœ‘bH bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP¢µˆñH bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP}OX”J‘ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPø $@N’ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPŸ]5+N“ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPì¬ZJ” bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPùUú;• b6 jet/accuracy_2_of_6"trainer/global_step2:JP•°ÃLJ– bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP4¿"´N— bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP㘪¤N˜ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÒ Œ(H™ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÙFq¥Hš bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPº çJ› bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPä§bö@œ b; particle/accuracy_1_of_1"trainer/global_step2:JP/äEJ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP@j¿óHž bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPŽ}JŸ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP¨ÆEdJ  bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPa0%ðN¡ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPð¦J¢ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPZàÄH;£ b6 jet/accuracy_3_of_6"trainer/global_step2:JP0«4J¤ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÝ.×­H¥ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP0òðJ¦ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPp*³H§ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPgÿñi;¨ b6 jet/accuracy_7_of_7"trainer/global_step2:JP ­”J© bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP'3™&Nª bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPâÍVN« bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP–5ªN¬ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP"Uæ;­ b6 jet/accuracy_1_of_5"trainer/global_step2:JP0éJ® bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÑ­tH¯ bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPåeh2J° bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP [€N± bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPË ÂN² bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP —YK@³ b; particle/accuracy_3_of_4"trainer/global_step2:JP'ì·cH´ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPQ¢XÝHµ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPH¶ bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPýÇbíJ· bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÅÐGóH¸ bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP•×RH¹ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÊ`uø;º b6 jet/accuracy_3_of_3"trainer/global_step2:JPœ‘ZÉ@» b; particle/accuracy_2_of_5"trainer/global_step2:JPÌ¿yŠJ¼ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP»¡š‹H½ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPò̈KJ¾ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPdõ^N¿ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¢œ"÷JÀ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPz[}QJÁ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPlÎÌ”H bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÀGWüNà bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP\¡‚58Ä b3 particle/f_score"trainer/global_step2:JP“³&ûHÅ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPFU@NÆ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP(kpINÇ bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¬'`U;È b6 jet/accuracy_5_of_5"trainer/global_step2:JPr%wJÉ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÏ^"žHÊ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPâEyHË bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPö{:JÌ bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP©¯þHÍ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP²PX6HÎ bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPaÔ´ˆHÏ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP'e1HÐ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPY`|ôHÑ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP)ìÞLJÒ bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPN\?JÓ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP5†±JÔ bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPDGÈ«HÕ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPw¾&îJÖ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPžCãÜH× bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPýIêNØ bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPXÉZHÙ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPš‘¯2HÚ bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP‘èÊHÛ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP?èõˆJÜ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP<ÁHÝ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPT7 ’NÞ bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPî³÷˜Jß bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP·ÍHNà bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP’ºcNJá bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPe ‚Â@â b; particle/accuracy_3_of_3"trainer/global_step2:JP‡Ül<ã b7 particle/specificity"trainer/global_step2:JPú9;ä b6 jet/accuracy_5_of_7"trainer/global_step2:JPŠ¥ùNå bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPIbUNæ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP<¤ÿ´9ç b4 particle/accuracy"trainer/global_step2:JP-ãMkJè bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP'ýÑ;é b6 loss/val_total_loss"trainer/global_step2:JP±8`½Hê bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPâ Jë bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPqt”²Nì bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÁ“lk;í b6 jet/accuracy_4_of_4"trainer/global_step2:JPí£ÕãJî bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP×èl¸Hï bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPØñ†7Hð bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPJm~Jñ bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP~¶ØgJò bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP¨ÞW/Jó bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPMÙÊNô bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPµhèÕNõ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP$jÅtJö bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰Ï!IH÷ bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP"X+Nø bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP1tNù bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPwˆWHú bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPИ©Jû bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP+'. Jü bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP~!«Ný bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¢ûêJþ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP?;Òí2ÿ - _wandbruntime‚1813  _runtime‚1813¯À;{'€ ï¥ ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0483240932226181 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14933894574642181 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5184369683265686 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6360324621200562 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2989446520805359 particle/accuracy_7_of_7‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9135802984237671 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0030141386669129133 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006605223286896944 particle/accuracy_4_of_6‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34767112135887146 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558467626571655 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12600956857204437 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3154762089252472 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6244106888771057 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8666666150093079 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 6validation_average_jet_accuracy‚0.3315674364566803 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7280701398849487 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16383598744869232 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5333333611488342 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1484929323196411 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968511462211609 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869133830070496 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3140730559825897 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.23159222304821014 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5357142686843872 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.17424243688583374 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17000295221805573 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2822854220867157 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 *jet/accuracy_1_of_1‚0.3387954831123352 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46898555755615234 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.07025130838155746 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33322638273239136 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.24499225616455078 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3014208972454071 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 $ _timestamp‚1.788482986054762e+09 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4768480956554413 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.33201056718826294 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23862341046333313 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.25901496410369873 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31902432441711426 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4885355830192566 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2809631824493408 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21883083879947662 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4209597706794739 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12940183281898499 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.18800155818462372 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09991995990276337 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1090121939778328 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4228110611438751 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918621182441711 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9864704012870789 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4561403691768646 _runtime‚1813.598561508 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2611759901046753 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.26474839448928833 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17686745524406433 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9851189851760864 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2527543008327484 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9961161017417908 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.005914347246289253 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2827518880367279 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3950679898262024 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0684523805975914 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10130719095468521 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.13507424294948578 *jet/accuracy_2_of_4‚0.2677804231643677 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.193764328956604 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.16604778170585632 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12110945582389832 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.28205129504203796 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2608906030654907 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0Ozô{k.49391433596611023 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2667464017868042 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990098237991333 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9884172677993774 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4925594925880432 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.04885057732462883 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19470162689685822 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9903072118759155 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.20000000298023224 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988292455673218 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9852015972137451 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05841648578643799 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24689576029777527 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05815748870372772 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_6‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2631816864013672 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34010255336761475 particle/accuracy_3_of_7‚1 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2364061176776886 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9736841917037964 8 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.10000000149011612 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015552995726466179 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5833333134651184 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7380952835083008 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9975185990333557 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1846383661031723 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9523809552192688 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23906926810741425 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.19589336216449738 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08217336982488632 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.47058823704719543 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.013519511558115482 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929468631744385 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.009452239610254765 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5093140602111816 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013095238246023655 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9911110997200012 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9878913164138794 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8128669261932373 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9900959730148315 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.17320862412452698 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt3FBt/event_purity‚0.10999999940395355 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7533819079399109 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18536394834518433 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10277777910232544 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20240937173366547 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2427094280719757 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5184230804443359 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21949166059494019 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.24979501962661743 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt2FBt/FRt_purity‚0.47435900568962097 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20735828578472137 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26816990971565247 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.6000000238418579 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.20857955515384674 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.25641024112701416 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 particle/accuracy_1_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0825350433588028 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_5‚0.33860665559768677 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2413400560617447 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.03842216357588768 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.18888889253139496 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21228572726249695 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.15198412537574768 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.5701754689216614 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2678526043891907 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916604161262512 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10185185074806213 particle/accuracy_1_of_1‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2975461483001709 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.01904761977493763 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.16731952130794525 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6538461446762085 +jet/accuracy_3_of_6‚0.02631578966975212 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5725806355476379 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9868420958518982 jet/accuracy_7_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8785086870193481 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5769230723381042 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03328213095664978 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 particle/accuracy_3_of_4‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12853746116161346 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3822175860404968 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.014999999664723873 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22372345626354218 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2982456386089325 ,jet/accuracy_3_of_3‚0.016024066135287285 particle/accuracy_2_of_5‚1 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1769547164440155 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.2625811994075775 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24036440253257751 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9210728406906128 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2799627184867859 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19611172378063202 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3011111319065094 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.14128068089485168 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9851827025413513 ?&Purity/0FRt1SRqqt2FBt/event_proportion‚0.008757560513913631 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9943277835845947 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9862963557243347 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.18963763117790222 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9479252099990845 jet/accuracy_5_of_7‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 (particle/accuracy‚0.8301254510879517 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 *loss/val_total_loss‚54.245695764656496 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.25151515007019043 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3142361044883728 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0029703062027692795 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2191609889268875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 >&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.19565249979496002 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05665638670325279 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3481481373310089 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4990655779838562 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12940183281898499 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.42248791456222534 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.28760582208633423 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3750000298023224 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.19304199516773224 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5108550786972046 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.25245776772499084 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.285388320684433 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32843393087387085 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9244444966316223 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.334013432264328 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008013312704861164 particle/accuracy_2_of_4‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.010013594292104244 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7111111283302307 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2666666805744171 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.28085190057754517 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 jet/accuracy_6_of_6‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9615384340286255 6 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.472222238779068 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6428571939468384 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4234391748905182 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9967741966247559 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.19434525072574615 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00625380827113986 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024406895972788334 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.16384129226207733 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.36728397011756897 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1527777761220932 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7105262875556946 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9479166865348816 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 *jet/accuracy_1_of_4‚0.7987643480300903 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.15135110914707184 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 -"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3125 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20893386006355286 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3750000298023224 jet/accuracy_4_of_6‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921156167984009 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 particle/accuracy_4_of_7‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.29251155257225037 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5472056269645691 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0065167807042598724 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003150201519019902 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_4_of_5‚1 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.261904776096344 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0818784311413765 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.13769842684268951 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9949876666069031 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982340335845947 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9895644783973694 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1875 particle/accuracy_3_of_6‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09224225580692291 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1598963439464569 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.18963763117790222 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5564625859260559 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2929958701133728 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25269821286201477 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2643522024154663 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.01693548448383808 jet/accuracy_4_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3750000298023224 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8968254327774048 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13647343218326569 particle/accuracy_2_of_3‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.25724637508392334 trainer/global_step‚603 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21804288029670715 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20852886140346527 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36440277099609375 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 _step‚48 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.925198495388031 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2995484471321106 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2777777910232544 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00766644487157464 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333432674408 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36988067626953125 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5155658721923828 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9946236610412598 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.45592594146728516 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9940475821495056 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6296037435531616 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2010914534330368 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18519410490989685 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.46666663885116577 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9983022212982178 ,particle/sensitivity‚0.19367773830890656 jet/accuracy_5_of_6‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5015802383422852 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12271346896886826 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.27475759387016296 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23972734808921814 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1964072734117508 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10913477092981339 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 epoch‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23593488335609436 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.16282643377780914 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.004314412362873554 9"Purity/*FRt2SRqqt1FBt/event_purity‚0.0128480838611722 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.999402642250061 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9943704009056091 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23028962314128876 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.261904776096344 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333432674408fƒ3w ó¥ ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1769547164440155 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.2625811994075775 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24036440253257751 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9210728406906128 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2799627184867859 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19611172378063202 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3011111319065094 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.14128068089485168 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9851827025413513 ?&Purity/0FRt1SRqqt2FBt/event_proportion‚0.008757560513913631 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9943277835845947 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9862963557243347 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.18963763117790222 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9479252099990845 jet/accuracy_5_of_7‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 (particle/accuracy‚0.8301254510879517 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 *loss/val_total_loss‚54.245695764656496 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.25151515007019043 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3142361044883728 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0029703062027692795 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2191609889268875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 >&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.19565249979496002 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05665638670325279 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3481481373310089 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4990655779838562 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12940183281898499 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.42248791456222534 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.28760582208633423 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3750000298023224 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.19304199516773224 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5108550786972046 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.25245776772499084 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.285388320684433 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32843393087387085 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9244444966316223 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.334013432264328 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008013312704861164 particle/accuracy_2_of_4‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt0SRqqt2FBt%Þ±!ƒ~/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.010013594292104244 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7111111283302307 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2666666805744171 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.28085190057754517 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 jet/accuracy_6_of_6‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9615384340286255 6 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.472222238779068 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6428571939468384 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4234391748905182 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9967741966247559 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.19434525072574615 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00625380827113986 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024406895972788334 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.16384129226207733 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.36728397011756897 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1527777761220932 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7105262875556946 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9479166865348816 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 *jet/accuracy_1_of_4‚0.7987643480300903 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.15135110914707184 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 -"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3125 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20893386006355286 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3750000298023224 jet/accuracy_4_of_6‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921156167984009 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 particle/accuracy_4_of_7‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.29251155257225037 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5472056269645691 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0065167807042598724 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003150201519019902 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_4_of_5‚1 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.261904776096344 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0818784311413765 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.13769842684268951 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9949876666069031 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982340335845947 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9895644783973694 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1875 particle/accuracy_3_of_6‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09224225580692291 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1598963439464569 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.18963763117790222 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5564625859260559 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2929958701133728 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25269821286201477 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2643522024154663 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.01693548448383808 jet/accuracy_4_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3750000298023224 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8968254327774048 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13647343218326569 particle/accuracy_2_of_3‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.25724637508392334 trainer/global_step‚603 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21804288029670715 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20852886140346527 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36440277099609375 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 _step‚48 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.925198495388031 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2995484471321106 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2777777910232544 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00766644487157464 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333432674408 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36988067626953125 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5155658721923828 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9946236610412598 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.45592594146728516 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9940475821495056 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6296037435531616 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2010914534330368 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18519410490989685 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.46666663885116577 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9983022212982178 ,particle/sensitivity‚0.19367773830890656 jet/accuracy_5_of_6‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5015802383422852 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12271346896886826 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.27475759387016296 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23972734808921814 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1964072734117508 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10913477092981339 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 epoch‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23593488335609436 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.16282643377780914 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.004314412362873554 9"Purity/*FRt2SRqqt1FBt/event_purity‚0.0128480838611722 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.999402642250061 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9943704009056091 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23028962314128876 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.261904776096344 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0483240932226181 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14933894574642181 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5184369683265686 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6360324621200562 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2989446520805359 particle/accuracy_7_of_7‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9135802984237671 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0030141386669129133 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006605223286896944 particle/accuracy_4_of_6‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34767112135887146 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558467626571655 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12600956857204437 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3154762089252472 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6244106888771057 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8666666150093079 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 6validation_average_jet_accuracy‚0.3315674364566803 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7280701398849487 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16383598744869232 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5333333611488342 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1484929323196411 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968511462211609 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869133830070496 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3140730559825897 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.23159222304821014 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5357142686843872 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.17424243688583374 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17000295221805573 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2822854220867157 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 *jet/accuracy_1_of_1‚0.3387954831123352 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46898555755615234 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.07025130838155746 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33322638273239136 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.24499225616455078 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3014208972454071 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 $ _timestamp‚1.788482986054762e+09 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4768480956554413 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.33201056718826294 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23862341046333313 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.25901496410369873 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31902432441711426 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4885355830192566 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2809631824493408 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21883083879947662 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4209597706794739 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12940183281898499 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.18800155818462372 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09991995990276337 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1090121939778328 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4228110611438751 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918621182441711 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9864704012870789 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4561403691768646 _runtime‚1813.598561508 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2611759901046753 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.26474839448928833 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17686745524406433 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9851189851760864 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2527543008327484 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9961161017417908 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.005914347246289253 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2827518880367279 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3950679898262024 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0684523805975914 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10130719095468521 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.13507424294948578 *jet/accuracy_2_of_4‚0.2677804231643677 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.193764328956604 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.16604778170585632 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12110945582389832 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.28205129504203796 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2608906030654907 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49391433596611023 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2667464017868042 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990098237991333 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9884172677993774 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4925594925880432 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.04885057732462883 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19470162689685822 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9903072118759155 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.20000000298023224 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988292455673218 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9852015972137451 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05841648578643799 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24689576029777527 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05815748870372772 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_6‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2631816864013672 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34010255336761475 particle/accuracy_3_of_7‚1 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2364061176776886 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9736841917037964 8 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.10000000149011612 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015552995726466179 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5833333134651184 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7380952835083008 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9975185990333557 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1846383661031723 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9523809552192688 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23906926810741425 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.19589336216449738 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08217336982488632 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.47058823704719543 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.013519511558115482 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929468631744385 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.009452239610254765 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5093140602111816 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013095238246023655 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9911110997200012 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9878913164138794 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8128669261932373 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9900959730148315 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.17320862412452698 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt3FBt/event_purity‚0.10999999940395355 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7533819079399109 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18536394834518433 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10277777910232544 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20240937173366547 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2427094280719757 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5184230804443359 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21949166059494019 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.24979501962661743 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt2FBt/FRt_purity‚0.47435900568962097 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20735828578472137 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26816990971565247 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.6000000238418579 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.20857955515384674 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.25641024112701416 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 particle/accuracy_1_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0825350433588028 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_5‚0.33860665559768677 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2413400560617447 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.03842216357588768 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.18888889253139496 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21228572726249695 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.15198412537574768 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.5701754689216614 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2678526043891907 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916604161262512 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10185185074806213 particle/accuracy_1_of_1‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2975461483001709 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.01904761977493763 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.16731952130794525 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6538461446762085 +jet/accuracy_3_of_6‚0.02631578966975212 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5725806355476379 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9868420958518982 jet/accuracy_7_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8785086870193481 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5769230723381042 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03328213095664978 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 particle/accuracy_3_of_4‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12853746116161346 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3822175860404968 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.014999999664723873 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22372345626354218 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2982456386089325 ,jet/accuracy_3_of_3‚0.016024066135287285 particle/accuracy_2_of_5‚1 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ("Purity/3FRt*SRqqt1FBt/event_purity‚00HDÝÛo‚ ‚(:… ««èÔñ¢ñß gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚35.0 gpu.0.powerWatts‚93.694' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7388 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# ‘ÐQ!gpu.0.uncorrectedMemoryErrors‚0\¥÷xeƒ ‚(:Ú ««èÔšù•à proc.cpu.threads‚17 network.recv‚584688 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 54.765625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 3902.44140625 disk.sda.out‚ 54.765625% proc.memory.availableMB‚ 805.90625% disk./.usageGB‚1161.8651847839355 disk.sda1.in‚0 disk.drbd1260.out‚ 0.24609375$ memory_percent‚95.08113861083984 network.sent‚ 1.299953e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3902.44140625# disk.nvme0c0n1.in‚ 3902.44140625# proc.memory.rssMB‚ 3913.05859375* proc.memory.percent‚23.883414268493652 cpu‚24.996694248701672( disk./.usagePercent‚77.9737474206729 disk.dm-0.out‚ 38202.2265625" disk.nvme0n1.out‚ 38202.2265625 disk.sda3.in‚0 disk.sda3.out‚ 54.765625$ disk.nvme0c0n1.out‚ 38202.2265625 disk.sda.in‚0rétU„ ‚(:Ê º«èÔ•Ýúß disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 3902.4453125 disk.nvme0n1.in‚ 3902.4453125 disk.nvme0n1.out‚ 46945.21875 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 55.12890625$ memory_percent‚99.96659755706787 proc.memory.rssMB‚5908% proc.memory.percent‚ 36.0595703125" disk.nvme0c0n1.in‚ 3902.4453125" disk.nvme0c0n1.out‚ 46945.21875 disk.dm-1.out‚ 55.12890625 disk.drbd1260.out‚ 0.24609375 disk.dm-0.out‚ 46945.21875& proc.memory.availableMB‚ 5.47265625 cpu‚24.99620098810721 proc.cpu.threads‚17% disk./.usageGB‚1170.9597053527832 disk.sda.out‚ 55.12890625 disk.dm-1.in‚0 disk.sda3.in‚0 network.sent‚ 1.401831e+06 network.recv‚590164) disk./.usagePercent‚78.58408832685927¢¿L"… ‚(:… º«èÔØè€á gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.871' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7742 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ûAi† ‚(:Þ É«èÔËó¢ß disk.sda3.in‚0* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 3902.4453125 disk.sda1.in‚0 disk.dm-1.out‚55.59375 disk.drbd1260.out‚0.265625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 3902.4453125& disk.nvme0n1.out‚48273.93310546875 disk.sda3.out‚55.59375# memory_percent‚68.6014175415039 network.sent‚ 1.408317e+06) disk./.usagePercent‚78.31511273097205( disk.nvme0c0n1.out‚48273.93310546875 disk.sda.out‚55.59375& proc.memory.availableMB‚ 5144.34375# proc.memory.rssMB‚ 3338.34765625 cpu‚24.999113279268798 network.recv‚595094$ disk./.usageGB‚1166.951774597168 disk.sda2.in‚0 disk.dm-0.in‚ 3902.4453125# disk.dm-0.out‚48273.93310546875‰¾ä8‘‡ ‚(:† É«èÔŽÀ‹à gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.006' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6012 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î3Îtˆ ‚(:é Ø«èÔû÷á# memory_percent‚68.6061143875122* proc.memory.percent‚20.375657081604004 network.recv‚599676 disk.sda3.in‚0 disk.sda.out‚ 56.02734375 disk.dm-1.out‚ 56.02734375 disk.drbd1260.in‚0 disk.dm-0.in‚ 3902.4453125# proc.memory.rssMB‚ 3338.34765625( disk./.usagePercent‚78.3154286442693 disk.drbd1260.out‚0.265625 disk.sda2.in‚0# disk.dm-0.out‚48519.39794921875& disk.nvme0n1.out‚48519.39794921875) proc.memory.availableMB‚ 5143.57421875 cpu‚25.00178335230162( disk.nvme0c0n1.out‚48519.39794921875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 3902.4453125 proc.cpu.threads‚17 network.sent‚ 1.414023e+06% disk./.usageGB‚1166.9564819335938 disk.sda3.out‚ 56.02734375" disk.nvme0c0n1.in‚ 3902.4453125 disk.sda.in‚0 disk.sda1.out‚0DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213092439052288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1982551424512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1978982595072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13651373 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16125061 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9304775 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204285199010277 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1979842563913 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213096033309696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211113481885184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211113456457216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213092439052288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1982551424512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1978982595072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13651373 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16125061 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9304775 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204285199010277 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1979842563913 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213096033309696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211113481885184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211113456457216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9318527 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43047695688704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1985546024448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1981977181696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204589860520091 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1982833076735 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213413618926080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13671629 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13671623 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1979266239195 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40504603359232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29794894 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204593452785599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16123271 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213413618926080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211428072901632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16149379 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5672317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119785472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40504574261248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16123271 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9319308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 trainer/global_step‚649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202610619708864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 % _timestamp‚1.7884831371693377e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29794894 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211428047473664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43047575903232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29821008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1985546024448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚781 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90687488 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29821008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886S:~ý&ž ÷M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213413618926080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211428072901632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16149379 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5672317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119785472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40504574261248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16123271 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9319308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 trainer/global_step‚649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202610619708864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 % _timestamp‚1.7884831371693377e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29794894 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211428047473664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43047575903232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29821008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1985546024448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚781 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90687488 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29821008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13671629 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9318527 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43047695688704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1985546024448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1981977181696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204589860520091 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1982833076735 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213413618926080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13671629 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13671623 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1979266239195 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40504603359232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29794894 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204593452785599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16123271 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99089408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17316931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228975341291520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17316931 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚217392498571664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14663879 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228975341291520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226843032707584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14663879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚811 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17343013 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226843007279616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2132308583936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2731180779520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43520322183680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219518301672985 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6077404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14663873 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3915707 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31980804 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2128739754496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228971747034112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43520351281664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761954816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2128739754496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99089408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17316931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228975341291520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17316931 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚217392498571664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14663879 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228975341291520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226843032707584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14663879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚811 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17343013 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226843007279616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2132308583936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2731180779520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43520322183680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219518301672985 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6077404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14663873 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3915707 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31980804 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2128739754496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228971747034112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43520351281664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 761954816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2128739754496 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17367331 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219822963182799 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2135303183872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32051460 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚217694144140896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17341223 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6085601 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14684123 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734924562944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2131734341120 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2735019444736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17341223 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3921188 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2131734341120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43581897157632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43581868059648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2128819041903 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚217694169568864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229292926907904 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10006789 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 YIDeviceStatsMonitor.on_train_batch_en.a Wd/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3921191 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14684123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979776 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14684129 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14684129 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17367331 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219822963182799 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2135303183872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32051460 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚217694144140896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17341223 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6085601 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14684123 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734924562944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2131734341120 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2735019444736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17341223 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3921188 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2131734341120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43581897157632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 761954816 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43581868059648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2128819041903 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚217694169568864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229292926907904 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10006789 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3921191 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 579526983687xpÅe2¿ - _wandbruntime‚2114  _runtime‚2114RÄÞXÀ Ò 2loss/SRqqt4/assignment_loss‚1.2090523690338941 1loss/FBt1/assignment_loss‚0.42943017067126577 .loss/FRt3/detection_loss‚2.519809093937945 1loss/SRqqt4/detection_loss‚1.0670000866118363 /loss/FBt2/detection_loss‚1.0914725823331606 epoch‚1 0loss/FRt2/assignment_loss‚3.4545504629611967 .loss/FRt4/detection_loss‚1.955880635637578 % _timestamp‚1.7884832870133705e+09 /loss/FRt1/detection_loss‚3.3616245771065736 1loss/SRqqt1/detection_loss‚2.9704025402301695 0loss/SRqqt3/detection_loss‚2.105165390274193 _step‚56 2loss/SRqqt2/assignment_loss‚2.6917916650050566 .loss/FBt1/detection_loss‚2.660631411605411 %loss/total_loss‚52.27859893895967 /loss/FRt4/assignment_loss‚1.785321233718376 2loss/SRqqt1/assignment_loss‚1.8626293231074402 1loss/FBt3/assignment_loss‚0.39920163815128484 1loss/SRqqt2/detection_loss‚2.6806626225772656 trainer/global_step‚699 /loss/FBt3/detection_loss‚2.1614827176457956 _runtime‚2114.554653524 /loss/FRt3/assignment_loss‚4.970818252705816 2loss/SRqqt3/assignment_loss‚1.6168152484335476 0loss/FBt4/assignment_loss‚0.2937116186616954 /loss/FBt4/detection_loss‚2.7043859197033777 /loss/FRt1/assignment_loss‚5.647896992854583 0loss/FBt2/assignment_loss‚0.2685835098001661 /loss/FRt2/detection_loss‚2.3702788761920397ŒDdÒ\Á Ö /loss/FBt3/detection_loss‚2.1614827176457956 _runtime‚2114.554653524 /loss/FRt3/assignment_loss‚4.970818252705816 2loss/SRqqt3/assignment_loss‚1.6168152484335476 0loss/FBt4/assignment_loss‚0.2937116186616954 /loss/FBt4/detection_loss‚2.7043859197033777 /loss/FRt1/assignment_loss‚5.647896992854583 0loss/FBt2/assignment_loss‚0.2685835098001661 /loss/FRt2/detection_loss‚2.3702788761920397 2loss/SRqqt4/assignment_loss‚1.2090523690338941 1loss/FBt1/assignment_loss‚0.42943017067126577 .loss/FRt3/detection_loss‚2.519809093937945 1loss/SRqqt4/detection_loss‚1.0670000866118363 /loss/FBt2/detection_loss‚1.0914725823331606 epoch‚1 0loss/FRt2/assignment_loss‚3.4545504629611967 .loss/FRt4/detection_loss‚1.955880635637578 % _timestamp‚1.7884832870133705e+09 /loss/FRt1/detection_loss‚3.3616245771065736 1loss/SRqqt1/detection_loss‚2.9704025402301695 0loss/SRqqt3/detection_loss‚2.105165390274193 _step‚56 2loss/SRqqt2/assignment_loss‚2.6917916650050566 .loss/FBt1/detection_loss‚2.660631411605411 %loss/total_loss‚52.27859893895967 /loss/FRt4/assignment_loss‚1.785321233718376 2loss/SRqqt1/assignment_loss‚1.8626293231074402 1loss/FBt3/assignment_loss‚0.39920163815128484 1loss/SRqqt2/detection_loss‚2.6806626225772656 trainer/global_step‚6998!oð@s ‚(:è ×­èÔòôÞá network.sent‚ 1.588871e+06) disk./.usagePercent‚78.37757780130407# disk.nvme0c0n1.in‚ 4494.86328125 disk.sda.out‚ 64.015625 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚99.96411800384521& proc.memory.availableMB‚ 5.87890625% disk./.usageGB‚1167.8825492858887& disk.nvme0n1.out‚54309.26513671875 disk.sda3.in‚0 disk.sda3.out‚ 64.015625( disk.nvme0c0n1.out‚54309.26513671875 disk.drbd1260.out‚ 0.31640625* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17 disk.dm-0.in‚ 4494.86328125! disk.nvme0n1.in‚ 4494.86328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 64.015625 disk.drbd1260.in‚0 cpu‚25.049727201568814 network.recv‚669002# disk.dm-0.out‚54309.26513671875 disk.sda.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3338.34765625ªœšé“à ‚(:ˆ ×­èÔëá£â gpu.0.gpu‚86.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚48.0 gpu.0.powerWatts‚320.591' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.1182 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o¡’6mÄ ‚(:â æ­èÔ³Ó˜à disk.sda2.out‚0$ memory_percent‚99.96120929718018# proc.memory.rssMB‚ 3338.34765625! disk.nvme0c0n1.in‚ 4632.390625 disk.sda.out‚ 64.421875 disk.drbd1260.in‚0 disk.sda2.in‚0* proc.memory.percent‚20.375657081604004 cpu‚25.052259332666313 proc.cpu.threads‚17) disk./.usagePercent‚78.37758420149244 disk.nvme0n1.in‚ 4632.390625 disk.sda3.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 6.35546875 network.sent‚ 1.675126e+06 disk.dm-0.in‚ 4632.390625# disk.dm-0.out‚54847.11669921875 disk.sda3.out‚ 64.421875( disk.nvme0c0n1.out‚54847.11669921875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 64.421875 network.recv‚674342% disk./.usageGB‚1167.8826446533203& disk.nvme0n1.out‚54847.11669921875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.316406258&z“Å ‚(:ˆ æ­èÔŸá gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚48.0 gpu.0.powerWatts‚248.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-¾oÆ ‚(:ä õ­èÔÕ€¶Þ network.sent‚ 1.681423e+06 disk.drbd1260.in‚0 disk.sda2.in‚0# disk.dm-0.out‚55195.31884765625 disk.sda3.in‚0 disk.sda.out‚ 65.16015625& proc.memory.availableMB‚ 6.34765625 proc.cpu.threads‚17$ disk./.usageGB‚1166.820468902588 disk.dm-1.out‚ 65.16015625 disk.dm-0.in‚ 4632.390625 disk.nvme0n1.in‚ 4632.390625 disk.sda3.out‚ 65.16015625( disk.nvme0c0n1.out‚55195.31884765625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚55195.31884765625 disk.sda1.in‚0" memory_percent‚99.961256980896# proc.memory.rssMB‚ 3338.34765625 cpu‚25.054087268502954 network.recv‚679141) disk./.usagePercent‚78.30630069563595 disk.drbd1260.out‚ 0.3203125! disk.nvme0c0n1.in‚ 4632.390625* proc.memory.percent‚20.375657081604004…²- “Ç ‚(:ˆ õ­èÔúªÒÞ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚250.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.0088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Õ¹jÈ ‚(:ß „®èÔ»ù«ß disk.sda1.out‚0 disk.dm-1.out‚ 65.57421875 disk.dm-0.in‚ 4632.390625 disk.nvme0n1.in‚ 4632.390625) disk./.usagePercent‚78.30630069563595 disk.sda.out‚ 65.57421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 65.57421875# proc.memory.rssMB‚ 3338.34765625 cpu‚25.05616259072262 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.3203125& disk.nvme0n1.out‚55387.77587890625 disk.sda3.in‚0# memory_percent‚99.9612808227539# proc.memory.availableMB‚6.34375* proc.memory.percent‚20.375657081604004! disk.nvme0c0n1.in‚ 4632.390625 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚55387.77587890625 proc.cpu.threads‚17 network.sent‚ 1.6873e+06 network.recv‚683614$ disk./.usageGB‚1166.820468902588( disk.nvme0c0n1.out‚55387.77587890625áÂÉ ‚(:’ „®èÔ¥Íß gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚50.0 gpu.0.powerWatts‚327.402' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.48039999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tª mÊ ‚(:â “®èÔ¹¤§ß' disk.nvme0c0n1.out‚55677.9521484375 disk.sda.in‚0 disk.sda.out‚ 65.94140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 65.94140625 disk.dm-1.in‚0 disk.dm-1.out‚ 65.94140625 disk.drbd1260.out‚ 0.32421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4632.390625" disk.dm-0.out‚55677.9521484375 disk.nvme0n1.in‚ 4632.390625% disk.nvme0n1.out‚55677.9521484375 disk.drbd1260.in‚0$ memory_percent‚99.96132850646973% proc.memory.availableMB‚ 6.3359375* proc.memory.percent‚20.375657081604004 cpu‚25.05812212940859 network.sent‚ 1.69239e+06 network.recv‚686964 disk.sda3.in‚0# proc.memory.rssMB‚ 3338.34765625 proc.cpu.threads‚17) disk./.usagePercent‚78.30472727333053% disk./.usageGB‚1166.7970237731934! disk.nvme0c0n1.in‚ 4632.390625;{ì’“Ë ‚(:ˆ “®èÔÀ Òß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚229.234' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.8468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P¬ÞoÌ ‚(:ä ¢®èÔö€¤á disk.dm-1.out‚ 66.3359375$ memory_percent‚99.96285438537598 disk.nvme0n1.in‚ 4793.6328125% disk.nvme0n1.out‚56133.8076171875 disk.sda3.in‚0 disk.sda3.out‚ 66.3359375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.32421875% proc.memory.availableMB‚ 6.0859375 proc.cpu.threads‚17 network.recv‚690334 disk.dm-0.in‚ 4793.6328125' disk.nvme0c0n1.out‚56133.8076171875 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 cpu‚25.060782124311977" disk.dm-0.out‚56133.8076171875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.697461e+06) disk./.usagePercent‚78.30472650530793% disk./.usageGB‚1166.7970123291016" disk.nvme0c0n1.in‚ 4793.6328125 disk.sda.out‚ 66.3359375 disk.sda1.in‚0NåZœÍ ‚(:‘ ¢®èÔ‹¦Ùâ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚43.0 gpu.0.powerWatts‚255.723' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.144600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨w*²mÎ ‚(:â ±®èÔ™äòÝ disk.sda3.out‚ 67.1171875 disk.sda2.out‚0 disk.dm-0.in‚ 4793.6328125$ memory_percent‚99.96161460876465 proc.cpu.threads‚17 disk.sda1.out‚0$ disk./.usageGB‚1166.720359802246& disk.nvme0n1.out‚56519.90771484375" disk.nvme0c0n1.in‚ 4793.6328125 disk.sda.in‚0 disk.sda.out‚ 67.1171875 disk.dm-1.in‚0* proc.memory.percent‚20.375657081604004( disk./.usagePercent‚78.2995822899149 disk.dm-1.out‚ 67.1171875# disk.dm-0.out‚56519.90771484375% proc.memory.availableMB‚ 6.2890625# proc.memory.rssMB‚ 3338.34765625 disk.sda1.in‚0 network.recv‚695083 disk.sda3.in‚0( disk.nvme0c0n1.out‚56519.90771484375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.328125 disk.sda2.in‚0 cpu‚25.063309757501347 network.sent‚ 1.70386e+06 disk.nvme0n1.in‚ 4793.6328125 WÃo“Ï ‚(:ˆ ±®èÔ¦¢•Þ gpu.0.gpu‚76.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚49.0 gpu.0.powerWatts‚334.216' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª”fhÐ ‚(:Ý À®èÔ¡¯¾ß disk.sda3.in‚0 disk.sda3.out‚67.4375 disk.sda.in‚0$ memory_percent‚99.96368885040283 proc.cpu.threads‚17 network.sent‚ 1.709758e+06 network.recv‚699408 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 4793.6328125( disk.nvme0c0n1.out‚56961.53173828125* proc.memory.percent‚20.375657081604004 cpu‚25.065038075174737 disk.drbd1260.in‚0# disk.dm-0.out‚56961.53173828125 disk.sda.out‚67.4375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 5.94921875# proc.memory.rssMB‚ 3338.34765625% disk./.usageGB‚1166.7991256713867 disk.drbd1260.out‚0.328125 disk.dm-0.in‚ 4793.6328125 disk.dm-1.out‚67.4375) disk./.usagePercent‚78.30486833348188 disk.nvme0n1.in‚ 4793.6328125& disk.nvme0n1.out‚56961.53173828125´ýçØ“Ñ ‚(:ˆ À®èÔÐåÐà gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚241.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GþðnÒ ‚(:ã Ï®èÔ†ûæÞ cpu‚25.067228338982055 network.sent‚ 1.714777e+06) disk./.usagePercent‚78.31015309701121$ disk./.usageGB‚1166.877872467041 disk.dm-1.out‚ 67.83203125 disk.sda2.in‚0" disk.dm-0.out‚57235.4052734375" disk.nvme0c0n1.in‚ 4793.6328125 disk.sda1.out‚0 network.recv‚702681' disk.nvme0c0n1.out‚57235.4052734375 disk.sda.out‚ 67.83203125 disk.sda1.in‚0" proc.memory.availableMB‚6.1875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.33203125 disk.nvme0n1.in‚ 4793.6328125% disk.nvme0n1.out‚57235.4052734375 disk.sda3.out‚ 67.83203125$ memory_percent‚99.96223449707031# proc.memory.rssMB‚ 3338.34765625 disk.sda2.out‚0 disk.dm-0.in‚ 4793.6328125 disk.sda3.in‚0 disk.sda.in‚0* proc.memory.percent‚20.375657081604004R¸¹‘Ó ‚(:† Ï®èÔã Ûß gpu.0.gpu‚79.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚114.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wëSû2Ô - _wandbruntime‚2241  _runtime‚2241zlÎpÅÕ ¿ trainer/global_step‚749 % _timestamp‚1.7884834135512102e+09 _runtime‚2241.093528142 _step‚57 & lr-AdamW/pg1‚0.0010118080726225207 & lr-AdamW/pg2‚0.0010118080726225207hÝÏÉÖ Ã % _timestamp‚1.7884834135512102e+09 _runtime‚2241.093528142 _step‚57 & lr-AdamW/pg1‚0.0010118080726225207 & lr-AdamW/pg2‚0.0010118080726225207 trainer/global_step‚7499œÊ±2× - _wandbruntime‚2241  _runtime‚2241¿ (ìüØ êO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2278948169329 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2282065743360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2922568891904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244851055015936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234754996600993 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15676373 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242572583529984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15676373 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49520184873984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 2241.09772258 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244851055015936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34237344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18534883 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242572558102016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2282065743360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4189907 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232476023003696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34211256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15676379 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6492682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46597645080064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚780 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚58 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244854649273344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15676379 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2275381331997 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242572583529984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242572558102016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2922661689856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234751404335693 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082« O†ô'Ù îO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2922661689856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234751404335693 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2278948169329 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2282065743360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2922568891904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244851055015936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234754996600993 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15676373 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242572583529984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15676373 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49520184873984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 2241.09772258 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244851055015936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34237344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18534883 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242572558102016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2282065743360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4189907 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232476023003696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34211256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15676379 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6492682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46597645080064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚780 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚58 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244854649273344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15676379 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2275381331997 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242572583529984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242572558102016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895936:!Fë2Ú - _wandbruntime‚2243  _runtime‚2243bg6Lú&Û ôM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15696623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4195391 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235056065845507 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6500944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34255798 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242887174546432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚779 _runtime‚2243.766626129 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2281491500544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2926501400064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18585283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49585692356096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242887149118464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232777694000896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2281491500544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10697114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2278371844611 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34281912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10696332 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245172234889728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245168640619008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2926408615424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245172234889728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235059658111015 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18585283 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18559175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34255798 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49585570473472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92784640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232777719428864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15696623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242887149118464 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2285060343296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚782 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46659190956032 _step‚59 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15696629 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788483416224451e+09üÄ(ÍÉÜ øM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2278371844611 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34281912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10696332 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245172234889728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245168640619008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2926408615424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245172234889728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235059658111015 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18585283 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18559175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34255798 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49585570473472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92784640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232777719428864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15696623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242887149118464 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2285060343296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚782 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46659190956032 _step‚59 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15696629 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788483416224451e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15696623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4195391 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235056065845507 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6500944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34255798 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242887174546432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚779 _runtime‚2243.766626129 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2281491500544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2926501400064 B8DeviceStatsMonitx[5or.on_train_batch_end/active.all.current‚26114 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18585283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49585692356096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242887149118464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232777694000896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2281491500544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10697114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665;Ú¾,ñ2Ý - _wandbruntime‚2243  _runtime‚2243˜¯ò˜\Þ Ö /loss/FRt4/detection_loss‚1.7322648269158822 1loss/SRqqt3/detection_loss‚2.5270006774049816 trainer/global_step‚749 0loss/FBt1/assignment_loss‚0.5450330489166486 1loss/FBt4/assignment_loss‚0.35655564409950913 /loss/FRt2/detection_loss‚2.1089990948525705 /loss/FBt2/detection_loss‚0.8006588285184519 /loss/FRt2/assignment_loss‚3.311698902858777 2loss/SRqqt4/assignment_loss‚1.2398841877318918 epoch‚1 _runtime‚2243.767668043 2loss/SRqqt1/assignment_loss‚2.1929052102689943 2loss/SRqqt3/assignment_loss‚2.0861579378445945 /loss/FRt1/detection_loss‚3.4689089671985522 $loss/total_loss‚52.2952593418047 /loss/FRt3/assignment_loss‚4.993651142245845 1loss/FBt2/assignment_loss‚0.26898139968460183 1loss/SRqqt1/detection_loss‚2.9364416593064866 1loss/SRqqt2/detection_loss‚2.4569388711175253 /loss/FBt1/detection_loss‚2.3458181104876776 % _timestamp‚1.7884834162264829e+09 /loss/FRt1/assignment_loss‚5.292158378137125 2loss/FBt3/assignment_loss‚0.048564512981883884 /loss/FRt3/detection_loss‚2.4972934440562597 1loss/SRqqt4/detection_loss‚1.0380662306845188 /loss/FBt3/detection_loss‚3.6438898970099056 .loss/FBt4/detection_loss‚2.404044436556952 0loss/FRt4/assignment_loss‚2.0736968720400775 2loss/SRqqt2/assignment_loss‚1.9256470608849858 _step‚60÷kI`ß Ú 1loss/SRqqt2/detection_loss‚2.4569388711175253 /loss/FBt1/detection_loss‚2.3458181104876776 % _timestamp‚1.7884834162264829e+09 /loss/FRt1/assignment_loss‚5.292158378137125 2loss/FBt3/assignment_loss‚0.048564512981883884 /loss/FRt3/detection_loss‚2.4972934440562597 1loss/SRqqt4/detection_loss‚1.0380662306845188 /loss/FBt3/detection_loss‚3.6438898970099056 .loss/FBt4/detection_loss‚2.404044436556952 0loss/FRt4/assignment_loss‚2.0736968720400775 2loss/SRqqt2/assignment_loss‚1.9256470608849858 _step‚60 /loss/FRt4/detection_loss‚1.7322648269158822 1loss/SRqqt3/detection_loss‚2.5270006774049816 trainer/global_step‚749 0loss/FBt1/assignment_loss‚0.5450330489166486 1loss/FBt4/assignment_loss‚0.35655564409950913 /loss/FRt2/detection_loss‚2.1089990948525705 /loss/FBt2/detection_loss‚0.8006588285184519 /loss/FRt2/assignment_loss‚3.311698902858777 2loss/SRqqt4/assignment_loss‚1.2398841877318918 epoch‚1 _runtime‚2243.767668043 2loss/SRqqt1/assignment_loss‚2.1929052102689943 2loss/SRqqt3/assignment_loss‚2.0861579378445945 /loss/FRt1/detection_loss‚3.4689089671985522 $loss/total_loss‚52.2952593418047 /loss/FRt3/assignment_loss‚4.993651142245845 1loss/FBt2/assignment_loss‚0.26898139968460183 1loss/SRqqt1/detection_loss‚2.9364416593064866DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2428500972037 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19752835 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2428254073344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249988099263701 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16688873 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4464107 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260730362997760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3114546556416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2431822902784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16688879 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3114641451520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4464104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19778917 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19752835 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52789456336896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884835548936894e+09 _step‚62 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258302134352384 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2424934134705 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19778917 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 _runtime‚2382.436179946 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6908413 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 trainer/global_step‚799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123993088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49674938878464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1`€¤T!÷ ïO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123993088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49674938878464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247559598291664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258302134352384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2428254073344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258302108924416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚806 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2431822902784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49674909780480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36441708 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260733957255168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260730362997760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11371711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94895104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2428500972037 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19752835 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2428254073344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249988099263701 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16688873 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4464107 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260730362997760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3114546556416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2431822902784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16688879 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3114641451520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4464104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19778917 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19752835 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52789456336896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884835548936894e+09 _step‚62 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258302134352384 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2424934134705 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19778917 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 _runtime‚2382.436179946 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6908413 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 trainer/global_step‚799>ûæý62ø - _wandbruntime‚2384  _runtime‚2384‡ÌRû&ù õM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19803235 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49736484754432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250292760773723 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2427924647319 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19803235 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2431248659968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52854980000768 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2431248659968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6916599 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6915802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36512364 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36486250 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49736455656448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2434817502720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19777127 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2431491484859 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4469591 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258616699940864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4469588 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261051542871552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 _step‚63 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚797 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11386190 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11385390 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16709123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258616699940864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2434817502720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19777127 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52854856020992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16709129 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247861243860896 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2431248659968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6916599 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6915802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36512364 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36486250 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49736455656448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2434817502720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19777127 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2431491484859 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4469591 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258616699940864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4469588 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261051542871552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 _step‚63 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚797 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11386190 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11385390 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16709123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258616699940864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2434817502720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19777127 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52854856020992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16709129 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247861243860896 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224?‘d|2û - _wandbruntime‚2384  _runtime‚23842ÕYü Ó % _timestamp‚1.7884835573093848e+09 0loss/FRt4/assignment_loss‚1.8215717795567634 1loss/SRqqt3/detection_loss‚2.0279345909754434 .loss/FBt3/detection_loss‚2.867788023182324 epoch‚1 0loss/FRt1/assignment_loss‚4.7853733521920665 2loss/SRqqt4/assignment_loss‚1.1196579816285521 .loss/FRt4/detection_loss‚1.866064177109645 /loss/FBt2/detection_loss‚0.9417665128343405 trainer/global_step‚799 1loss/SRqqt1/assignment_loss‚2.249550176518304 2loss/SRqqt2/assignment_loss‚2.2819186548391976 1loss/FBt2/assignment_loss‚0.22215859216386982 /loss/FRt2/detection_loss‚2.2478353210601463 1loss/SRqqt2/detection_loss‚2.7269184754954443 0loss/FBt3/assignment_loss‚0.4437985344214082 1loss/FBt4/assignment_loss‚0.37889114055118017 /loss/FRt1/detection_loss‚3.0639502549612962 /loss/FBt4/detection_loss‚2.5963847120602925 .loss/FBt1/detection_loss‚2.570194807317522 &loss/total_loss‚50.520201292920795 0loss/FRt2/assignment_loss‚3.2112598727984603 /loss/FRt3/assignment_loss‚4.956058220018314 0loss/SRqqt3/assignment_loss‚1.90952460151366 /loss/FRt3/detection_loss‚2.4369803621799133 1loss/SRqqt4/detection_loss‚0.9639095224142075 1loss/FBt1/assignment_loss‚0.30009990666012715 0loss/SRqqt1/detection_loss‚2.530611720468317 _runtime‚2384.850960038 _step‚64PÍž]ý × .loss/FBt3/detection_loss‚2.867788023182324 epoch‚1 0loss/FRt1/assignment_loss‚4.7853733521920665 2loss/SRqqt4/assignment_loss‚1.1196579816285521 .loss/FRt4/detection_loss‚1.866064177109645 /loss/FBt2/detection_loss‚0.9417665128343405 trainer/global_step‚799 1loss/SRqqt1/assignment_loss‚2.249550176518304 2loss/SRqqt2/assignment_loss‚2.2819186548391976 1loss/FBt2/assignment_loss‚0.22215859216386982 /loss/FRt2/detection_loss‚2.2478353210601463 1loss/SRqqt2/detection_loss‚2.7269184754954443 0loss/FBt3/assignment_loss‚0.4437985344214082 1loss/FBt4/assignment_loss‚0.37889114055118017 /loss/FRt1/detection_loss‚3.0639502549612962 /loss/FBt4/detection_loss‚2.5963847120602925 .loss/FBt1/detection_loss‚2.570194807317522 &loss/total_loss‚50.520201292920795 0loss/FRt2/assignment_loss‚3.2112598727984603 /loss/FRt3/assignment_loss‚4.956058220018314 0loss/SRqqt3/assignment_loss‚1.90952460151366 /loss/FRt3/detection_loss‚2.4369803621799133 1loss/SRqqt4/detection_loss‚0.9639095224142075 1loss/FBt1/assignment_loss‚0.30009990666012715 0loss/SRqqt1/detection_loss‚2.530611720468317 _runtime‚2384.850960038 _step‚64 % _timestamp‚1.7884835573093848e+09 0loss/FRt4/assignment_loss‚1.8215717795567634 1loss/SRqqt3/detection_loss‚2.0279345909754434@QxÑrþ ‚(:ç å¯èÔÅšµà# disk.nvme0c0n1.in‚ 5255.84765625( disk.nvme0c0n1.out‚60746.32080078125 disk.dm-1.in‚0$ memory_percent‚99.96211528778076& proc.memory.availableMB‚ 6.20703125* proc.memory.percent‚20.375657081604004 network.sent‚ 1.851661e+06 network.recv‚745082 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1167.8825912475586 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3515625# disk.dm-0.out‚60746.32080078125! disk.nvme0n1.in‚ 5255.84765625& disk.nvme0n1.out‚60746.32080078125 disk.sda3.out‚ 72.078125 disk.sda.out‚ 72.078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 72.078125 disk.sda2.out‚0 disk.dm-0.in‚ 5255.84765625# proc.memory.rssMB‚ 3338.34765625 cpu‚25.084637107733233) disk./.usagePercent‚78.377580617386951oOä‘ÿ ‚(:† å¯èÔÏü¸à gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚40.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cþ‘s€ ‚(:è ô¯èÔ»¹‡á disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 5255.84765625# disk.nvme0c0n1.in‚ 5255.84765625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 72.58203125 disk.sda2.out‚0$ memory_percent‚99.96223449707031" proc.memory.availableMB‚6.1875 network.recv‚751384) disk./.usagePercent‚78.37758061738695 disk.sda3.in‚0 disk.sda.out‚ 72.58203125 cpu‚25.08603008794911# disk.dm-0.out‚61189.05126953125( disk.nvme0c0n1.out‚61189.05126953125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.3515625# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17 network.sent‚ 1.938811e+06% disk./.usageGB‚1167.8825912475586 disk.dm-0.in‚ 5255.84765625& disk.nvme0n1.out‚61189.05126953125 disk.sda3.out‚ 72.58203125¦R’Ÿ“ ‚(:ˆ ô¯èÔ²»œâ gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚121.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦hýi‚ ‚(:Þ ƒ°èÔקîß# disk.nvme0c0n1.out‚ 61329.578125 disk.sda.in‚0 disk.drbd1260.in‚0 disk.dm-0.out‚ 61329.578125! disk.nvme0n1.in‚ 5255.84765625# memory_percent‚99.9645471572876* proc.memory.percent‚20.375657081604004 cpu‚25.087243305213022 network.sent‚ 1.943904e+06) disk./.usagePercent‚78.37758061738695% disk./.usageGB‚1167.8825912475586 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 73.13671875! disk.nvme0n1.out‚ 61329.578125& proc.memory.availableMB‚ 5.80859375 network.recv‚754924# disk.nvme0c0n1.in‚ 5255.84765625 disk.sda.out‚ 73.13671875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.35546875 disk.sda2.in‚0 disk.dm-0.in‚ 5255.84765625 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 73.13671875# proc.memory.rssMB‚ 3338.34765625 proc.cpu.threads‚17à¥Ö­“ƒ ‚(:ˆ ƒ°èÔ΂‘â gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚246.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.E¢“„ ‚(:ˆ ’°èÔÁòÝ gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚42.0 gpu.0.powerWatts‚119.037' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8074 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³A«f… ‚(:à ’°èÔ¯”úÞ* proc.memory.percent‚20.375657081604004 cpu‚25.088435696467393 network.recv‚758314) disk./.usagePercent‚78.34841598307877% disk.nvme0c0n1.out‚61700.16796875 disk.dm-1.out‚ 73.5703125 disk.dm-0.out‚61700.16796875! disk.nvme0n1.in‚ 5255.84765625$ memory_percent‚99.96151924133301# proc.memory.rssMB‚ 3338.34765625 disk.drbd1260.in‚0% disk./.usageGB‚1167.4480171203613# disk.nvme0c0n1.in‚ 5255.84765625 disk.sda.in‚0 disk.sda.out‚ 73.5703125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5255.84765625# disk.nvme0n1.out‚61700.16796875 disk.sda3.in‚0 disk.sda3.out‚ 73.5703125% proc.memory.availableMB‚ 6.3046875 proc.cpu.threads‚173ksAT disk.drbd1260.out‚ 0.35546875 network.sent‚ 1.94891e+06 disk.sda1.out‚0¢:Ês† ‚(:è ¡°èÔø„ß' disk.nvme0c0n1.out‚62289.1455078125% disk.nvme0n1.out‚62289.1455078125 disk.sda3.in‚0$ memory_percent‚99.96168613433838 cpu‚25.08959440294929% disk./.usageGB‚1167.4499130249023 disk.sda.out‚ 73.93359375 disk.sda1.in‚0 disk.dm-1.out‚ 73.93359375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5387.26171875" disk.dm-0.out‚62289.1455078125 network.sent‚ 1.955272e+06 network.recv‚763082# disk.nvme0c0n1.in‚ 5387.26171875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.359375 disk.sda2.out‚0) disk./.usagePercent‚78.34854321882328! disk.nvme0n1.in‚ 5387.26171875 disk.sda3.out‚ 73.93359375& proc.memory.availableMB‚ 6.27734375# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17+X“‡ ‚(:ˆ ¡°èÔ‡¹Éß gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚234.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.8662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äx‰¶nˆ ‚(:ã °°èÔ‰³©í$ memory_percent‚99.96025562286377* proc.memory.percent‚20.375657081604004 cpu‚25.090642049555605' disk.nvme0c0n1.out‚62692.6064453125 disk.sda2.out‚0" disk.dm-0.out‚62692.6064453125% disk.nvme0n1.out‚62692.6064453125 disk.sda3.in‚0 network.sent‚ 1.962954e+06 disk.sda.in‚0 disk.sda.out‚ 74.515625 disk.dm-1.out‚ 74.515625 disk.dm-0.in‚ 5387.26171875& proc.memory.availableMB‚ 6.51171875# proc.memory.rssMB‚ 3338.34765625 proc.cpu.threads‚17 network.recv‚767281% disk./.usageGB‚1167.4260559082031# disk.nvme0c0n1.in‚ 5387.26171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.359375 disk.sda2.in‚0) disk./.usagePercent‚78.34694214770414 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 5387.26171875 disk.sda3.out‚ 74.515625¦ßí0›‰ ‚(: °°èÔÄ©€ð gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.178' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.43560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Ñ£MtŠ ‚(:é ¿°èÔ©žÔÞ disk.sda2.out‚0# memory_percent‚99.9617338180542 network.recv‚770786 disk.sda3.in‚0( disk.nvme0c0n1.out‚63111.60302734375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.36328125* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17 network.sent‚ 1.96614e+06) disk./.usagePercent‚78.34851710605479 disk.dm-0.in‚ 5387.26171875# disk.nvme0c0n1.in‚ 5387.26171875 disk.sda1.in‚0& proc.memory.availableMB‚ 6.26953125# proc.memory.rssMB‚ 3338.34765625% disk./.usageGB‚1167.4495239257812# disk.dm-0.out‚63111.60302734375 disk.sda.out‚ 74.9296875 disk.dm-1.in‚0 disk.dm-1.out‚ 74.9296875 cpu‚25.091685033718726! disk.nvme0n1.in‚ 5387.26171875& disk.nvme0n1.out‚63111.60302734375 disk.sda3.out‚ 74.9296875 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0›ŒŸû“‹ ‚(:ˆ ¿°èÔô×âß gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚44.0 gpu.0.powerWatts‚238.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÂ5+qŒ ‚(:æ ΰèԚϥß$ disk./.usageGB‚1167.449504852295 disk.sda1.out‚0 disk.drbd1260.out‚ 0.36328125& disk.nvme0n1.out‚63362.02490234375 network.sent‚ 1.97135e+06# disk.nvme0c0n1.in‚ 5387.26171875( disk.nvme0c0n1.out‚63362.02490234375 disk.dm-0.in‚ 5387.26171875 disk.sda3.in‚0$ proc.memory.availableMB‚6.015625 proc.cpu.threads‚17 network.recv‚774371 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚63362.02490234375$ memory_percent‚99.96328353881836* proc.memory.percent‚20.375657081604004 disk.sda.in‚0 disk.sda.out‚ 75.1328125 disk.dm-1.out‚ 75.1328125 disk.sda2.in‚0! disk.nvme0n1.in‚ 5387.26171875 disk.sda3.out‚ 75.1328125# proc.memory.rssMB‚ 3338.34765625 cpu‚25.09279308215069) disk./.usagePercent‚78.34851582601712Ä»lê“ ‚(:ˆ ΰèÔ©©¿à gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚231.383' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.2766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ox'@lŽ ‚(:á Ý°èÔ¬Ô¢Þ% disk./.usageGB‚1167.2020225524902" disk.nvme0c0n1.in‚ 5498.4296875( disk.nvme0c0n1.out‚63696.04345703125 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3671875& disk.nvme0n1.out‚63696.04345703125 disk.sda3.in‚0 disk.sda3.out‚ 75.546875 disk.sda.out‚ 75.546875 disk.sda2.in‚0# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚779084) disk./.usagePercent‚78.33190708122574 disk.sda1.in‚0 disk.dm-1.out‚ 75.546875 disk.sda2.out‚0 disk.dm-0.in‚ 5498.4296875$ memory_percent‚99.96175765991211 disk.nvme0n1.in‚ 5498.4296875# disk.dm-0.out‚63696.04345703125$ proc.memory.availableMB‚6.265625 cpu‚25.09392073769241 network.sent‚ 1.977606e+06hÕUÿ’ ‚(:‡ Ý°èÔ¤ªÇß gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚238.335' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ¥2 - _wandbruntime‚2519  _runtime‚2519éÂMÞÅ‘ ¿ & lr-AdamW/pg1‚0.0010085148115510634 & lr-AdamW/pg2‚0.0010085148115510634 trainer/global_step‚849 % _timestamp‚1.7884836916563425e+09 _runtime‚2519.197764523 _step‚65Šþ_ŠÉ’ à & lr-AdamW/pg1‚0.0010085148115510634 & lr-AdamW/pg2‚0.0010085148115510634 trainer/global_step‚849 % _timestamp‚1.7884836916563425e+09 _runtime‚2519.197764523 _step‚65AŠÒ‚]2“ - _wandbruntime‚2519  _runtime‚2519€6uèñ'” ëO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 _runtime‚2519.201008128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274031659746816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265221201926409 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 _step‚66 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17701379 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276609670979584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 % _timestamp‚1.7884836916578107e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17701373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2581580062208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265217609661109 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 trainer/global_step‚849 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12061635 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20996869 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20996869 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274031685174784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38672160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2578011232768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3306949431296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4738304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2574486937413 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274031659746816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12060867 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7322563 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276609670979584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3306858730496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56059182108160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20970787 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 764176896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578053774745 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274031685174784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52752203578880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚262643148151664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2574486937413 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5308 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274031659746816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12060867 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7322563 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276609670979584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3306858730496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56059182108160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20970787 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 764176896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578053774745 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274031685174784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52752203578880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚262643148151664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276609670979584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 % _timestamp‚1.7884836916578107e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17701373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2581580062208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265217609661109 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 trainer/global_step‚849 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12061635 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20996869 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20996869 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274031685174784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38672160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2578011232768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3306949431296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4738304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577477450027 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38716702 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12074602 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3310796510208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17721623 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884836944463854e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38742816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56124455277568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retriZ´¯Z es‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7331582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20995079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2581044287567 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119785472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚67 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚2521.989003431 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276930850853376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274346276191232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17721623 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2584574662144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12075373 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚771 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265522271170923 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3310705822720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 764176896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262944793720896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2581005819392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52813749454848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262944819148864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276930850853376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584574662144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4743791 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56124575063040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38742816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17721629 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20995079 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21021187 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276927256582656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274346250763264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17721629 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52813778552832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 trainer/global_step‚849£ÿÿ&š ùM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262944793720896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2581005819392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52813749454848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262944819148864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276930850853376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584574662144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4743791 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56124575063040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57952698368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38742816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17721629 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20995079 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21021187 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276927256582656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274346250763264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17721629 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52813778552832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 trainer/global_step‚849 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21021187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4743788 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5308 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577477450027 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4555014144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38716702 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12074602 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3310796510208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17721623 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884836944463854e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38742816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56124455277568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7331582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20995079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2581044287567 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119785472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚67 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚2521.989003431 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276930850853376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274346276191232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17721623 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2584574662144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12075373 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚771 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265522271170923 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3310705822720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 764176896C40V?2› - _wandbruntime‚2521  _runtime‚2521"}JRœ Ì _runtime‚2521.991128087 _step‚68 0loss/FBt4/assignment_loss‚0.5048291053520783 1loss/SRqqt2/detection_loss‚2.7792143243200638 /loss/FBt3/detection_loss‚2.3903556737032803 0loss/FBt3/assignment_loss‚0.3016599327216461 % _timestamp‚1.7884836944494188e+09 2loss/SRqqt1/assignment_loss‚1.4661041420352245 1loss/SRqqt2/assignment_loss‚2.466451324084226 0loss/FBt1/assignment_loss‚0.3760034039065176 .loss/FRt1/detection_loss‚3.300226637295314 &loss/total_loss‚50.407061358579824 epoch‚1 2loss/SRqqt4/assignment_loss‚1.2103973191885933 1loss/SRqqt4/detection_loss‚1.1033913501882338 trainer/global_step‚849 .loss/FRt3/assignment_loss‚4.92210328578949 /loss/FRt4/assignment_loss‚2.167842093160597 1loss/FBt2/assignment_loss‚0.22917683919169154 .loss/FBt1/detection_loss‚2.801181925194604 /loss/FBt2/detection_loss‚0.9157825035750953 /loss/FRt2/assignment_loss‚3.210502486228943 .loss/FRt3/detection_loss‚2.537754243367339 /loss/FRt4/detection_loss‚1.8144231558603756 0loss/SRqqt3/assignment_loss‚1.54814141216438 .loss/FRt2/detection_loss‚2.166302769277245 1loss/SRqqt1/detection_loss‚2.6636006949203357 .loss/FBt4/detection_loss‚2.595417022705078 /loss/FRt1/assignment_loss‚4.747421108946508 1loss/SRqqt3/detection_loss‚2.18877860540296969eÈýV Ð /loss/FBt2/detection_loss‚0.9157825035750953 /loss/FRt2/assignment_loss‚3.210502486228943 .loss/FRt3/detection_loss‚2.537754243367339 /loss/FRt4/detection_loss‚1.8144231558603756 0loss/SRqqt3/assignment_loss‚1.54814141216438 .loss/FRt2/detection_loss‚2.166302769277245 1loss/SRqqt1/detection_loss‚2.6636006949203357 .loss/FBt4/detection_loss‚2.595417022705078 /loss/FRt1/assignment_loss‚4.747421108946508 1loss/SRqqt3/detection_loss‚2.1887786054029696 _runtime‚2521.991128087 _step‚68 0loss/FBt4/assignment_loss‚0.5048291053520783 1loss/SRqqt2/detection_loss‚2.7792143243200638 /loss/FBt3/detection_loss‚2.3903556737032803 0loss/FBt3/assignment_loss‚0.3016599327216461 % _timestamp‚1.7884836944494188e+09 2loss/SRqqt1/assignment_loss‚1.4661041420352245 1loss/SRqqt2/assignment_loss‚2.466451324084226 0loss/FBt1/assignment_loss‚0.3760034039065176 .loss/FRt1/detection_loss‚3.300226637295314 &loss/total_loss‚50.407061358579824 epoch‚1 2loss/SRqqt4/assignment_loss‚1.2103973191885933 1loss/SRqqt4/detection_loss‚1.1033913501882338 trainer/global_step‚849 .loss/FRt3/assignment_loss‚4.92210328578949 /loss/FRt4/assignment_loss‚2.167842093160597 1loss/FBt2/assignment_loss‚0.22917683919169154 .loss/FBt1/detection_loss‚2.801181925194604DuÑÉHž ‚(:’ û°èÔÝöóà gpu.0.gpu‚99.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚49.0 gpu.0.powerWatts‚328.599' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.71979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ/þuŸ ‚(:ê û°èÔöü—ã disk.sda1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚20.375657081604004 disk.sda.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 6.74609375# proc.memory.rssMB‚ 3338.34765625" disk.nvme0c0n1.in‚ 5498.4296875 network.recv‚788728 disk.sda2.out‚0 disk.dm-0.in‚ 5498.4296875# disk.dm-0.out‚64719.89013671875 disk.sda.out‚ 76.57421875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 76.57421875( disk.nvme0c0n1.out‚64719.89013671875 disk.dm-1.out‚ 76.57421875 disk.drbd1260.out‚ 0.37109375$ memory_percent‚99.95882511138916 cpu‚25.09619008556267& disk.nvme0n1.out‚64719.89013671875 network.sent‚ 2.071482e+06) disk./.usagePercent‚78.26686452703898% disk./.usageGB‚1166.2328414916992 disk.nvme0n1.in‚ 5498.4296875EN¾Âž  ‚(:“ Š±èÔóóåÝ gpu.0.gpu‚63.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚220.157' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.031400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é¸}Uf¡ ‚(:Û Š±èÔ§ Éß! disk.dm-0.out‚65163.361328125 disk.sda3.in‚0& disk.nvme0c0n1.out‚65163.361328125 network.recv‚793656) disk./.usagePercent‚78.23686198007123 disk.dm-1.in‚0 disk.dm-1.out‚76.90625 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚65163.361328125 disk.sda3.out‚76.90625 disk.dm-0.in‚ 5498.4296875 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚99.96190071105957% proc.memory.availableMB‚ 6.2421875 proc.cpu.threads‚17 network.sent‚ 2.136264e+06 disk.nvme0n1.in‚ 5498.4296875" disk.nvme0c0n1.in‚ 5498.4296875 disk.sda.out‚76.90625 disk.sda1.out‚0# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 cpu‚25.097046429908506% disk./.usageGB‚1165.7857818603516 disk.drbd1260.out‚ 0.37109375 disk.sda2.out‚0Ùé –n¢ ‚(:ã ™±èÔ ÇÁá# disk.nvme0c0n1.in‚ 5621.28515625) disk./.usagePercent‚78.23686479615411% disk./.usageGB‚1165.7858238220215! disk.nvme0n1.in‚ 5621.28515625 disk.sda3.in‚0 disk.sda3.out‚ 77.1796875* proc.memory.percent‚20.375657081604004 network.sent‚ 2.140544e+06 disk.dm-0.in‚ 5621.28515625' disk.nvme0c0n1.out‚65259.2900390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 77.1796875 disk.drbd1260.in‚0 network.recv‚798356" disk.dm-0.out‚65259.2900390625% disk.nvme0n1.out‚65259.2900390625 disk.sda2.out‚0$ memory_percent‚99.96035099029541& proc.memory.availableMB‚ 6.49609375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 77.1796875 disk.drbd1260.out‚0.375 disk.sda2.in‚0# proc.memory.rssMB‚ 3338.34765625 cpu‚25.097971842455348Z,ž£ ‚(:“ ™±èÔØ–€ä gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚251.452' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.290400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìò]µa¤ ‚(:Ö ¨±èÔñŸõà# proc.memory.rssMB‚ 3338.34765625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚77.875 disk.sda1.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 6.48046875 disk.sda.in‚0 disk.dm-1.out‚77.875 disk.dm-0.in‚ 5621.28515625" disk.dm-0.out‚65354.6298828125$ memory_percent‚99.96044635772705* proc.memory.percent‚20.375657081604004 cpu‚25.09874775558914 disk.dm-1.in‚0 network.sent‚ 2.148237e+06) disk./.usagePercent‚78.23686351611643% disk./.usageGB‚1165.7858047485352 disk.sda3.in‚0' disk.nvme0c0n1.out‚65354.6298828125 disk.sda.out‚77.875 disk.drbd1260.in‚0 network.recv‚802197# disk.nvme0c0n1.in‚ 5621.28515625 disk.drbd1260.out‚0.375 disk.sda2.out‚0! disk.nvme0n1.in‚ 5621.28515625% disk.nvme0n1.out‚65354.6298828125 kÙ “¥ ‚(:ˆ ¨±èÔö¢¹á gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚228.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.7872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´_2@`¦ ‚(:Õ ·±èÔª£äÝ disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚99.96192455291748# proc.memory.rssMB‚ 3338.34765625$ disk.nvme0c0n1.out‚ 65666.6328125 disk.sda1.out‚0 disk.dm-1.out‚78.3125 disk.sda2.in‚0 disk.dm-0.out‚ 65666.6328125* proc.memory.percent‚20.375657081604004 cpu‚25.099939918792163) disk./.usagePercent‚78.23686095604108 disk.sda.out‚78.3125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.3828125 disk.sda3.out‚78.3125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 5621.28515625! disk.nvme0n1.in‚ 5621.28515625" disk.nvme0n1.out‚ 65666.6328125 disk.sda3.in‚0& proc.memory.availableMB‚ 6.23828125 proc.cpu.threads‚17 network.sent‚ 2.150719e+06 network.recv‚805882% disk./.usageGB‚1165.7857666015625# disk.nvme0c0n1.in‚ 5621.28515625ÏN£ž§ ‚(:“ ·±èÔöœ«à gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚249.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.857600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K_󪞨 ‚(:“ ƱèÔËÂÐÝ gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.061' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.612199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00³Hn© ‚(:ã ƱèÔÆ’‹Þ disk.sda2.out‚0! disk.nvme0n1.in‚ 5621.28515625$ memory_percent‚99.96044635772705) disk./.usagePercent‚78.23016021483988$ disk./.usageGB‚1165.685920715332 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 78.62109375 disk.dm-0.in‚ 5621.28515625# disk.nvme0n1.out‚65955.37109375* proc.memory.percent‚20.375657081604004 network.recv‚809257 disk.sda3.out‚ 78.62109375% disk.nvme0c0n1.out‚65955.37109375 disk.drbd1260.out‚ 0.3828125# proc.memory.rssMB‚ 3338.34765625 cpu‚25.100865056363123 proc.cpu.threads‚17 network.sent‚ 2.155731e+06# disk.nvme0c0n1.in‚ 5621.28515625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.out‚65955.37109375& proc.memory.availableMB‚ 6.48046875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 78.62109375 disk.sda2.in‚0Éý›Örª ‚(:ç Õ±èÔÉæëã) disk./.usagePercent‚78.23917372810541 disk.sda3.out‚ 78.9453125# disk.nvme0c0n1.in‚ 5756.10546875( disk.nvme0c0n1.out‚66299.83740234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 78.9453125# proc.memory.rssMB‚ 3338.34765625 disk.dm-0.in‚ 5756.10546875! disk.nvme0n1.in‚ 5756.10546875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.38671875 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.availableMB‚6.46875 cpu‚25.10176677964483 network.recv‚814504# disk.dm-0.out‚66299.83740234375& disk.nvme0n1.out‚66299.83740234375 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.96051788330078* proc.memory.percent‚20.375657081604004 network.sent‚ 2.163895e+06% disk./.usageGB‚1165.8202285766602 disk.sda.out‚ 78.9453125 proc.cpu.threads‚17±Uˆ« ‚(:’ Õ±èÔÁˆŠä gpu.0.gpu‚21.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚159.242' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.848399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ÆUˆg¬ ‚(:Ü ä±èÔÜç»Þ disk.sda.out‚ 79.4296875& proc.memory.availableMB‚ 6.45703125# proc.memory.rssMB‚ 3338.34765625 cpu‚25.10287377297105 proc.cpu.threads‚17) disk./.usagePercent‚78.22943161739744# disk.dm-0.out‚66669.88037109375 disk.nvme0n1.in‚ 5763.6875 disk.sda3.in‚0 disk.sda2.out‚0$ memory_percent‚99.96058940887451* proc.memory.percent‚20.375657081604004 network.sent‚ 2.16788e+06 network.recv‚818811$ disk./.usageGB‚1165.675064086914 disk.dm-0.in‚ 5763.6875& disk.nvme0n1.out‚66669.88037109375 disk.nvme0c0n1.in‚ 5763.6875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 79.4296875( disk.nvme0c0n1.out‚66669.88037109375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 79.4296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.38671875 disk.sda2.in‚0=A®x“­ ‚(:ˆ ä±èÔ£òÙß gpu.0.gpu‚94.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚213.448' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.6896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·$š2® - _wandbruntime‚2649  _runtime‚2649ˆX¢Å¯ ¿ trainer/global_step‚899 % _timestamp‚1.7884838214599943e+09 _runtime‚2649.001441746 _step‚69 & lr-AdamW/pg1‚0.0010062390514057379 & lr-AdamW/pg2‚0.0010062390514057379~!É° à _runtime‚2649.001441746 _step‚69 & lr-AdamW/pg1‚0.0010062390514057379 & lr-AdamW/pg2‚0.0010062390514057379 trainer/global_step‚899 % _timestamp‚1.7884838214599943e+09E/T'¹2± - _wandbruntime‚2649  _runtime‚2649m”@‘I ² èO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12750165 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5012504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788483821461308e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280454304589117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 _runtime‚ 2649.0060201 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59328013646848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59328141834240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280450712323817 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277726672583696 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55829497377280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55829526475264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2731337221632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22188739 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 WGDeviceSu¥tatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292488978961408 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18713879 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22188739 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2731337221632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12749352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2727768392192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 _step‚70 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292492573218816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40902612 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292492573218816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7737658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22214821 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22214821 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292488978961408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40928700 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3498516269568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289761210569216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18713873 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18713879 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚810 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3498615358976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727768392192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289761210569216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277726698011664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18713873 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289761235997184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚813 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99089408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289761235997184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5012507 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2724039740121 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 trainer/global_step‚899 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 772303360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224ƒ;yxò'³ ìO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18713879 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚810 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3498615358976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57952698368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727768392192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289761210569216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277726698011664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18713873 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289761235997184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚813 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99089408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5308 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289761235997184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5012507 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2724039740121 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4555014144 trainer/global_step‚899 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1705473536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 772303360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12750165 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5012504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4555014144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788483821461308e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280454304589117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2172 _runtime‚ 2649.0060201 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59328013646848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2172 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57952698368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59328141834240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280450712323817 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277726672583696 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55829497377280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55829526475264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2731337221632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22188739 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5308 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292488978961408 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18713879 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22188739 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2731337221632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12749352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57952698368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2727768392192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 _step‚70 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4555014144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292492573218816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40902612 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292492573218816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7737658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22214821 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22214821 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292488978961408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40928700 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3498516269568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289761210569216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18713873 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345Fì!Ms2´ - _wandbruntime‚2651  _runtime‚2651ÇQû&µ õM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55891072351232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40947154 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2734331821568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5017991 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18734129 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22239139 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884838238679757e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3502431675904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96978944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 772303360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18734129 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40973268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚815 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7745133 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290075827013632 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278028369008864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18734123 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2730762978816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22213031 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59393504027136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12763939 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280758966099139 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40973268 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18734123 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7745948 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280755373833631 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2734331821568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3502334696960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5017988 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12763121 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290075801585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚818 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ôom–¶ ùM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40973268 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18734123 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7745948 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280755373833631 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2734331821568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3502334696960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57952698368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5017988 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4555014144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12763121 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2172 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290075801585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚818 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55891072351232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40947154 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.sma_!Åæll_pool.current‚ 3568842752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2734331821568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5017991 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18734129 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22239139 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884838238679757e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3502431675904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4555014144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96978944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 772303360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18734129 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5308 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5308 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5308 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40973268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚815 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7745133 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57952698368 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2172 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2172 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290075827013632 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278028369008864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18734123 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2730762978816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22213031 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59393504027136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1705473536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12763939 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280758966099139 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0G˜VÔ92· - _wandbruntime‚2651  _runtime‚2651*åçU¸ Ï 1loss/SRqqt3/assignment_loss‚1.891886576868688 .loss/FRt3/detection_loss‚2.487056465917512 % _timestamp‚1.7884838238717005e+09 .loss/FRt4/detection_loss‚1.802749301351252 .loss/FBt4/detection_loss‚2.367485974322666 %loss/total_loss‚49.55989624840136 2loss/SRqqt2/assignment_loss‚1.8851936623454093 _step‚72 /loss/FRt3/assignment_loss‚4.313079644190638 1loss/FBt3/assignment_loss‚0.38852058153818636 .loss/FBt1/detection_loss‚2.114400427592428 epoch‚1 _runtime‚2651.412966067 0loss/FRt2/assignment_loss‚3.1153767000545156 1loss/FBt1/assignment_loss‚0.46227123591023056 /loss/FRt2/detection_loss‚2.0751654124734076 /loss/FBt3/detection_loss‚3.1295818721546844 trainer/global_step‚899 /loss/FRt1/assignment_loss‚4.626399970636135 0loss/FRt4/assignment_loss‚1.9923395722631425 2loss/SRqqt4/assignment_loss‚1.2660350640686677 0loss/SRqqt4/detection_loss‚1.115499883792546 1loss/FBt2/assignment_loss‚0.21859093295247922 0loss/FBt4/assignment_loss‚0.3358970791566456 0loss/SRqqt2/detection_loss‚2.568983244895935 0loss/SRqqt3/detection_loss‚2.427980684026887 /loss/FBt2/detection_loss‚1.0058762665298044 1loss/SRqqt1/assignment_loss‚2.159100622915831 /loss/FRt1/detection_loss‚3.4206104249489018 1loss/SRqqt1/detection_loss‚2.3898146474947697Åþ&ÞY¹ Ó 2loss/SRqqt2/assignment_loss‚1.8851936623454093 _step‚72 /loss/FRt3/assignment_loss‚4.313079644190638 1loss/FBt3/assignment_loss‚0.38852058153818636 .loss/FBt1/detection_loss‚2.114400427592428 epoch‚1 _runtime‚2651.412966067 0loss/FRt2/assignment_loss‚3.1153767000545156 1loss/FBt1/assignment_loss‚0.46227123591023056 /loss/FRt2/detection_loss‚2.0751654124734076 /loss/FBt3/detection_loss‚3.1295818721546844 trainer/global_step‚899 /loss/FRt1/assignment_loss‚4.626399970636135 0loss/FRt4/assignment_loss‚1.9923395722631425 2loss/SRqqt4/assignment_loss‚1.2660350640686677 0loss/SRqqt4/detection_loss‚1.115499883792546 1loss/FBt2/assignment_loss‚0.21859093295247922 0loss/FBt4/assignment_loss‚0.3358970791566456 0loss/SRqqt2/detection_loss‚2.568983244895935 0loss/SRqqt3/detection_loss‚2.427980684026887 /loss/FBt2/detection_loss‚1.0058762665298044 1loss/SRqqt1/assignment_loss‚2.159100622915831 /loss/FRt1/detection_loss‚3.4206104249489018 1loss/SRqqt1/detection_loss‚2.3898146474947697 1loss/SRqqt3/assignment_loss‚1.891886576868688 .loss/FRt3/detection_loss‚2.487056465917512 % _timestamp‚1.7884838238717005e+09 .loss/FRt4/detection_loss‚1.802749301351252 .loss/FBt4/detection_loss‚2.367485974322666 %loss/total_loss‚49.55989624840136H¸þãÐ[º ‚(:Ð ó±èÔ…ÇÅÞ disk.nvme0n1.in‚ 5763.6875 disk.sda3.out‚79.8125 network.recv‚822147# disk.dm-0.out‚66718.75927734375& disk.nvme0n1.out‚66718.75927734375 disk.sda3.in‚0# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 cpu‚25.10386363527225 proc.cpu.threads‚17 network.sent‚ 2.172985e+06) disk./.usagePercent‚78.22975341886797( disk.nvme0c0n1.out‚66718.75927734375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚79.8125 disk.drbd1260.out‚0.390625 disk.sda2.out‚0$ disk./.usageGB‚1165.679859161377 disk.sda2.in‚0# memory_percent‚99.9610424041748% proc.memory.availableMB‚ 6.3828125 disk.nvme0c0n1.in‚ 5763.6875 disk.sda.out‚79.8125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 5763.6875G-©“» ‚(:ˆ ó±èÔÀŠÇß gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚46.0 gpu.0.powerWatts‚249.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.9612 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Éô/H¼ ‚(:’ ‚²èÔÖøß gpu.0.gpu‚42.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚251.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.397000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 §ÀØ_½ ‚(:Ô ‚²èÔò¸°à disk.sda1.in‚0 disk.dm-1.out‚80.15625 disk.drbd1260.out‚0.390625# disk.dm-0.out‚66945.65771484375 disk.nvme0n1.in‚ 5763.6875% disk./.usageGB‚1165.7802543640137 disk.nvme0c0n1.in‚ 5763.6875 disk.sda1.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚66945.65771484375 disk.sda3.in‚0$ memory_percent‚99.96118545532227# proc.memory.rssMB‚ 3338.34765625 network.sent‚ 2.261275e+06 disk.dm-1.in‚0 disk.dm-0.in‚ 5763.6875 proc.cpu.threads‚17 network.recv‚827372 disk.sda.out‚80.15625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚80.15625$ proc.memory.availableMB‚6.359375* proc.memory.percent‚20.375657081604004 cpu‚25.10489998803408) disk./.usagePercent‚78.23649102515411( disk.nvme0c0n1.out‚66945.65771484375 disk.sda.in‚01m†Žh¾ ‚(:Ý ‘²èÔ‰­“ã disk.drbd1260.out‚ 0.39453125# disk.dm-0.out‚67506.07080078125 disk.sda3.out‚ 80.5078125 disk.sda.out‚ 80.5078125 disk.dm-1.in‚0 disk.sda2.out‚0# memory_percent‚99.9598741531372& proc.memory.availableMB‚ 6.57421875# proc.memory.rssMB‚ 3338.34765625 disk.sda.in‚0 disk.dm-0.in‚ 5763.6875( disk.nvme0c0n1.out‚67506.07080078125 disk.sda1.out‚0 disk.dm-1.out‚ 80.5078125* proc.memory.percent‚20.375657081604004$ disk./.usageGB‚1164.941421508789 disk.nvme0c0n1.in‚ 5763.6875 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚25.105635877107435 proc.cpu.threads‚17 network.sent‚ 2.267663e+06 network.recv‚832342) disk./.usagePercent‚78.18019624841234 disk.nvme0n1.in‚ 5763.6875& disk.nvme0n1.out‚67506.07080078125 disk.sda3.in‚0 disk.drbd1260.in‚0BŽï“¿ ‚(:ˆ ‘²èÔ½Íóã gpu.0.gpu‚40.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚41.0 gpu.0.powerWatts‚115.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¼UŸvÀ ‚(:ë  ²èÔ°ì¬ß disk.nvme0n1.in‚ 5925.6015625" disk.nvme0c0n1.in‚ 5925.6015625 disk.sda.in‚0 disk.dm-1.out‚ 81.06640625 disk.drbd1260.out‚ 0.39453125 proc.cpu.threads‚17 disk.dm-0.in‚ 5925.6015625 disk.sda3.in‚0( disk.nvme0c0n1.out‚67982.77783203125 disk.dm-1.in‚0$ memory_percent‚99.96144771575928& proc.memory.availableMB‚ 6.31640625# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 network.recv‚836402) disk./.usagePercent‚78.19014214111758% disk./.usageGB‚1165.0896224975586# disk.dm-0.out‚67982.77783203125& disk.nvme0n1.out‚67982.77783203125 disk.sda3.out‚ 81.06640625 disk.drbd1260.in‚0 cpu‚25.106848655456982 network.sent‚ 2.271606e+06 disk.sda.out‚ 81.06640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0‡“=)“Á ‚(:ˆ  ²èÔ±úÅà gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚252.747' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.5494 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê‚6äs ‚(:è ¯²èÔõÍ¥á& proc.memory.availableMB‚ 6.43359375# disk.dm-0.out‚68197.50830078125 disk.sda2.out‚0$ memory_percent‚99.96073246002197# proc.memory.rssMB‚ 3338.34765625 proc.cpu.threads‚17 network.sent‚ 2.276792e+06 network.recv‚840076& disk.nvme0n1.out‚68197.50830078125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 81.82421875 cpu‚25.1078500003769% disk./.usageGB‚1165.0896224975586 disk.dm-0.in‚ 5925.6015625 disk.nvme0n1.in‚ 5925.6015625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 5925.6015625 disk.sda.out‚ 81.82421875 disk.drbd1260.in‚0* proc.memory.percent‚20.375657081604004) disk./.usagePercent‚78.19014214111758 disk.sda3.out‚ 81.82421875( disk.nvme0c0n1.out‚68197.50830078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.3984375 disk.sda2.in‚0À à ‚(:’ ¯²èÔÁîà gpu.0.gpu‚98.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚49.0 gpu.0.powerWatts‚337.958' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.59160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º”MsÄ ‚(:è ¾²èÔúíîÞ( disk.nvme0c0n1.out‚68396.00439453125 disk.dm-1.out‚ 82.22265625 disk.sda3.in‚0$ memory_percent‚99.96230602264404& proc.memory.availableMB‚ 6.17578125 cpu‚25.10865566998868) disk./.usagePercent‚78.20745132252263 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5925.6015625 disk.nvme0n1.in‚ 5925.6015625 network.sent‚ 2.281867e+06$ disk./.usageGB‚1165.347541809082 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.3984375# disk.dm-0.out‚68396.00439453125& disk.nvme0n1.out‚68396.00439453125 disk.sda.out‚ 82.22265625 disk.sda3.out‚ 82.22265625# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 proc.cpu.threads‚17 network.recv‚843690" disk.nvme0c0n1.in‚ 5925.6015625óqñŒÅ ‚(:… ¾²èÔ÷óß gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚258.05' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.61 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 DÆ‚aÆ ‚(:Ö ͲèÔ‰™…Þ network.sent‚ 2.288067e+06" disk.nvme0c0n1.in‚ 5925.6015625 disk.dm-1.in‚0# memory_percent‚99.9638557434082 proc.cpu.threads‚17 network.recv‚848166% disk./.usageGB‚1165.3475914001465 disk.sda.in‚0 disk.sda.out‚82.78125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚68512.55078125 disk.sda1.out‚0 disk.dm-1.out‚82.78125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.40234375 disk.dm-0.in‚ 5925.6015625 disk.nvme0n1.in‚ 5925.6015625# disk.nvme0n1.out‚68512.55078125 disk.sda3.in‚0) disk./.usagePercent‚78.20745465062058% disk.nvme0c0n1.out‚68512.55078125 disk.sda3.out‚82.78125$ proc.memory.availableMB‚5.921875# proc.memory.rssMB‚ 3338.34765625* proc.memory.percent‚20.375657081604004 cpu‚25.109777375734264ƒõøO“Ç ‚(:ˆ ͲèÔ÷Í„Þ gpu.0.gpu‚59.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚248.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô è“È ‚(:ˆ ܲèÔ铯Þ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚45.0 gpu.0.powerWatts‚244.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ÁgðfÉ ‚(:Û ܲèÔˆ·èÞ) disk./.usagePercent‚78.24113372178756 disk.dm-0.out‚ 68570.796875! disk.nvme0n1.out‚ 68570.796875 disk.sda3.out‚ 83.2109375# disk.nvme0c0n1.out‚ 68570.796875 disk.sda.out‚ 83.2109375 disk.dm-1.in‚0# proc.memory.rssMB‚ 3338.34765625! disk.nvme0n1.in‚ 6059.53515625 disk.dm-1.out‚ 83.2109375 disk.sda2.out‚0$ memory_percent‚99.96387958526611* proc.memory.percent‚20.375657081604004 network.sent‚ 2.294029e+06 disk.dm-0.in‚ 6059.53515625# disk.nvme0c0n1.in‚ 6059.53515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.40234375 disk.sda2.in‚0 cpu‚25.11059283508165 network.recv‚852742% disk./.usageGB‚1165.8494338989258 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 5.91796875 proc.cpu.threads‚17”æÓsÊ ‚(:è ë²èÔÙËùå disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.40625! disk.nvme0n1.in‚ 6059.53515625% disk.nvme0n1.out‚69038.9404296875 disk.sda3.out‚ 83.68359375$ memory_percent‚99.96087551116943 network.sent‚ 2.301034e+06 network.recv‚856431 disk.sda1.out‚0 disk.dm-1.out‚ 83.68359375 disk.sda2.out‚0# proc.memory.rssMB‚ 3338.34765625 cpu‚25.111721698135305 proc.cpu.threads‚17) disk./.usagePercent‚78.24780681417506% disk./.usageGB‚1165.9488677978516' disk.nvme0c0n1.out‚69038.9404296875 disk.sda1.in‚0 disk.dm-0.in‚ 6059.53515625& proc.memory.availableMB‚ 6.41015625* proc.memory.percent‚20.375657081604004# disk.nvme0c0n1.in‚ 6059.53515625 disk.sda.out‚ 83.68359375 disk.sda2.in‚0" disk.dm-0.out‚69038.9404296875 disk.sda3.in‚09k8“Ë ‚(:ˆ ë²èÔù­°æ gpu.0.gpu‚85.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚68.11042785644531* gpu.0.memoryAllocatedBytes‚ 58506412032 gpu.0.temp‚47.0 gpu.0.powerWatts‚299.236' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š·B’2Ì - _wandbruntime‚2782  _runtime‚2782Ù…ßúÄÍ ¾ trainer/global_step‚949 $ _timestamp‚1.788483954825797e+09 _runtime‚2782.367416097 _step‚73 & lr-AdamW/pg1‚0.0010035462884062092 & lr-AdamW/pg2‚0.0010035462884062092!Ù…ðÈÎ Â $ _timestamp‚1.788483954825797e+09 _runtime‚2782.367416097 _step‚73 & lr-AdamW/pg1‚0.0010035462884062092 & lr-AdamW/pg2‚0.0010035462884062092 trainer/global_step‚949IîÀÛØ2Ï - _wandbruntime‚2782  _runtime‚2782Ï:ö·ò'Ð ìO KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2173 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305490761391616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3689992913408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305490761391616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2873592542829 _step‚74 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4557111296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚834 trainer/global_step‚949 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19726379 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58906820273664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2782.371018878 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8149087 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292810247871664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19726379 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19726373 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295683814986525 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43133064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308368286943232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3689891726848 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2173 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23406691 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130284544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4557111296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877525551616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292810222443696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305490786819584 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62596682902528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57954795520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295687407251825 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5286704 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308371881200640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13435791 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1711103488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2881094381056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2881094381056 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101186560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23432773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8149921 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58906791175680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4557111296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57954795520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308368286943232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57954795520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2173 % _timestamp‚1.7884839548273592e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305490786819584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 772303360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19726373ïïqÉ  Ñ ðO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23432773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8149921 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58906791175680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4557111296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57954795520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308368286943232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57954795520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2173 % _timestamp‚1.7884839548273592e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305490786819584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 772303360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19726373 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2173 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305490761391616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3689992913408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 E;DeviceStatsMonitor.on_train_batch_start/allocation.a€±ÅVll.peak‚31929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305490761391616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2873592542829 _step‚74 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4557111296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚834 trainer/global_step‚949 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19726379 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58906820273664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2782.371018878 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8149087 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292810247871664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19726379 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19726373 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295683814986525 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43133064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308368286943232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3689891726848 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2173 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23406691 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130284544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4557111296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877525551616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292810222443696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305490786819584 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62596682902528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57954795520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295687407251825 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5286704 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308371881200640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13435791 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1711103488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2881094381056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2881094381056 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101186560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0JñÐhS2Ò - _wandbruntime‚2784  _runtime‚2784ã6ëü&Ó öM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚308689466817024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚308685872546304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19746629 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23457091 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884839571261842e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2876583055443 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5292188 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4557111296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57954795520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43177606 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305805352408064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130271232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3693693037568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23457091 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2880520138240 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚833 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62662160360448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2884088980992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2880520138240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43203720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2173 trainer/global_step‚949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58968366149632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2880149892983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5309 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2173 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305805377836032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23430983 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101173248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23430983 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57954795520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43203720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13450263 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1717394944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295988476496339 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13449427 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2173 _runtime‚2784.668521242 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43177606 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚836 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62662030089216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293111918868864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19746629 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5292191 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8158072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295992068761847 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚308689466817024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19746623 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5309 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2173 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305805377836032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23430983 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101173248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23430983 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57954795520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43203720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13450263 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1717394944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295988476496339 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13449427 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2173 _runtime‚2784.668521242 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43177606 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚836 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62662030089216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293111918868864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19746629 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5292191 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8158072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295992068761847 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚308689466817024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19746623 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884839571261842e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2876583055443 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5292188 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4557111296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57954795520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43177606 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305805352408064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130271232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3693693037568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23457091 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2880520138240 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚833 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62662160360448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2884088980992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2880520138240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43203720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2173 trainer/global_step‚949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58968366149632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2880149892983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321220337641984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚828 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14121156 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5560907 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4557111296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3027282711040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚825 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65865084749824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚78 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 _runtime‚2921.105829705 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324247594925056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1717394944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884840935638344e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45389604 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14120328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310920509914533 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65864956562432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61984114072064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61984084974080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57954795520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20738873 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321220312214016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324247594925056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8559424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3023145345537 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4557111296 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2173 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3030851540480 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20738879 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324247594925056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8559424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3023145345537 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4557111296 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2173 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3030851540480 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20738879 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321220337641984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚828 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14121156 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5560907 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4557111296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3027282711040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚825 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65865084749824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚78 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 _runtime‚2921.105829705 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324247594925056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1717394944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884840935638344e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45389604 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14120328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310920509914533 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65864956562432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61984114072064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61984084974080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57954795520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20738873 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321220312214016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65930336580096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 _step‚79 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321534928658432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308195468728864 trainer/global_step‚999 % _timestamp‚1.7884840960950003e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321534903230464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3030277297664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45434172 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14134006 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45434172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20759123 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5309 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57954795520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8568424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321534903230464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚809 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5566391 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62045659948032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1717394944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311225171424555 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3033846140416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3033846140416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45408058 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45408058 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24675043 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321534928658432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324568774798848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57954795520 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2173 _runtime‚2923.640223002 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96978944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5309 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 785935872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3029702695691 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65930462657024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3884705730048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4557111296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324568774798848 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2173 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24648935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308195443300896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752’¿cvÿ&ò ùM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚809 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5566391 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62045659948032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1717394944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311225171424555 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3033846140416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3033846140416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45408058 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45408058 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24675043 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321534928658432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324568774798848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57954795520 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2173 _runtime‚2923.640223002 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96978944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5309 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 785935872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3029702695691 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65930462657024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3884705730048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4557111296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324568774798848 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2173 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24648935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308195443300896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62045630850048 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65930336580096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 _step‚79 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321534928658432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308195468728864 trainer/global_step‚999 % _timestamp‚1.7884840960950003e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321534903230464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3030277297664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45434172 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14134006 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45434172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20759123 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5309 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57954795520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8568424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321534903230464ObÃë³2ó - _wandbruntime‚2923  _runtime‚2923#—sXô Ò /loss/FRt2/assignment_loss‚2.659930108853106 /loss/FRt3/assignment_loss‚4.168120103428163 1loss/FBt3/assignment_loss‚0.47243893338815757 /loss/FBt3/detection_loss‚2.4822888981412956 2loss/SRqqt3/assignment_loss‚1.6765291931460033 /loss/FBt1/detection_loss‚2.1653383802622557 % _timestamp‚1.7884840961023233e+09 _runtime‚2923.644100972 2loss/SRqqt2/assignment_loss‚1.6309774952630203 2loss/SRqqt4/assignment_loss‚0.8775918724946677 0loss/FBt2/assignment_loss‚0.2392262605386457 epoch‚1 _step‚80 0loss/FRt4/assignment_loss‚1.6396943045928416 0loss/FBt4/assignment_loss‚0.1946373651995104 .loss/FRt1/detection_loss‚3.203233337523986 /loss/FBt4/detection_loss‚2.0484298142519863 2loss/SRqqt1/assignment_loss‚1.8331617986162503 /loss/FRt2/detection_loss‚1.9180867130745132 /loss/FRt4/detection_loss‚1.8085571331656085 /loss/SRqqt1/detection_loss‚2.81181172813688 1loss/SRqqt2/detection_loss‚2.7748288470837803 &loss/total_loss‚46.175902711572945 /loss/FRt3/detection_loss‚2.5455164934497283 0loss/SRqqt3/detection_loss‚2.182406073486483 /loss/FBt2/detection_loss‚0.7966309742135024 trainer/global_step‚999 /loss/FRt1/assignment_loss‚4.524754312573647 1loss/FBt1/assignment_loss‚0.44205156425925907 0loss/SRqqt4/detection_loss‚1.079661006429656}®%\õ Ö 0loss/SRqqt3/detection_loss‚2.182406073486483 /loss/FBt2/detection_loss‚0.7966309742135024 trainer/global_step‚999 /loss/FRt1/assignment_loss‚4.524754312573647 1loss/FBt1/assignment_loss‚0.44205156425925907 0loss/SRqqt4/detection_loss‚1.079661006429656 /loss/FRt2/assignment_loss‚2.659930108853106 /loss/FRt3/assignment_loss‚4.168120103428163 1loss/FBt3/assignment_loss‚0.47243893338815757 /loss/FBt3/detection_loss‚2.4822888981412956 2loss/SRqqt3/assignment_loss‚1.6765291931460033 /loss/FBt1/detection_loss‚2.1653383802622557 % _timestamp‚1.7884840961023233e+09 _runtime‚2923.644100972 2loss/SRqqt2/assignment_loss‚1.6309774952630203 2loss/SRqqt4/assignment_loss‚0.8775918724946677 0loss/FBt2/assignment_loss‚0.2392262605386457 epoch‚1 _step‚80 0loss/FRt4/assignment_loss‚1.6396943045928416 0loss/FBt4/assignment_loss‚0.1946373651995104 .loss/FRt1/detection_loss‚3.203233337523986 /loss/FBt4/detection_loss‚2.0484298142519863 2loss/SRqqt1/assignment_loss‚1.8331617986162503 /loss/FRt2/detection_loss‚1.9180867130745132 /loss/FRt4/detection_loss‚1.8085571331656085 /loss/SRqqt1/detection_loss‚2.81181172813688 1loss/SRqqt2/detection_loss‚2.7748288470837803 &loss/total_loss‚46.175902711572945 /loss/FRt3/detection_loss‚2.5455164934497283P)Èobö ‚(:× ´èÔû²ªß# disk./.usageGB‚1166.82661819458 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 6377.578125 disk.sda3.in‚0 disk.sda3.out‚87.5625$ memory_percent‚99.96495246887207% proc.memory.availableMB‚ 5.7421875 disk.dm-1.out‚87.5625 disk.dm-0.in‚ 6377.578125& disk.nvme0n1.out‚72165.63525390625 cpu‚25.12100039671032 proc.cpu.threads‚17 network.sent‚ 2.435416e+06) disk./.usagePercent‚78.30671337978116 disk.sda.out‚87.5625 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚72165.63525390625! disk.nvme0c0n1.in‚ 6377.578125( disk.nvme0c0n1.out‚72165.63525390625 network.recv‚897630 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.42578125" proc.memory.rssMB‚ 3338.3515625* proc.memory.percent‚20.375680923461914ÙŽ|a“÷ ‚(:ˆ ´èÔôžÈá gpu.0.gpu‚92.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.451' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä—9iø ‚(:Þ ´èÔßÖØâ disk.drbd1260.out‚ 0.42578125 disk.dm-0.in‚ 6377.578125# disk.dm-0.out‚72642.27587890625 disk.sda3.in‚0 disk.sda3.out‚ 87.953125$ memory_percent‚99.96204376220703# proc.memory.availableMB‚6.21875 disk.dm-1.out‚ 87.953125( disk.nvme0c0n1.out‚72642.27587890625 disk.sda2.out‚0& disk.nvme0n1.out‚72642.27587890625" proc.memory.rssMB‚ 3338.3515625 cpu‚25.121748852831075% disk./.usageGB‚1167.1786346435547 disk.sda.out‚ 87.953125 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 6377.578125* proc.memory.percent‚20.375680923461914 proc.cpu.threads‚17 network.sent‚ 2.524332e+06 network.recv‚903762! disk.nvme0c0n1.in‚ 6377.578125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚78.33033749903332ˆ(ôBù ‚(:’ ´èÔ…äÂâ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚45.0 gpu.0.powerWatts‚198.688' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚39.73759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ˆ¯ú ‚(:’ Ÿ´èÔ“Á÷Ý gpu.0.gpu‚99.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚49.0 gpu.0.powerWatts‚323.276' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.65520000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿àÛò\û ‚(:Ñ Ÿ´èÔô­ Þ disk.dm-1.out‚ 88.453125 disk.drbd1260.in‚0$ memory_percent‚99.96359348297119" proc.memory.rssMB‚ 3338.3515625 cpu‚25.122790697018967 network.sent‚ 2.529422e+06) disk./.usagePercent‚78.25937809871519 disk.sda3.out‚ 88.453125" disk.nvme0c0n1.out‚ 73085.40625 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚907553% disk./.usageGB‚1166.1212882995605 disk.nvme0n1.in‚ 6495.8671875 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 5.96484375* proc.memory.percent‚20.375680923461914 disk.nvme0n1.out‚ 73085.40625" disk.nvme0c0n1.in‚ 6495.8671875 disk.sda.in‚0 disk.drbd1260.out‚ 0.4296875 disk.sda2.in‚0 disk.dm-0.in‚ 6495.8671875 disk.dm-0.out‚ 73085.40625 disk.sda.out‚ 88.453125ÀWâ¨bü ‚(:× ®´èÔù©Àß disk.nvme0n1.in‚ 6495.8671875$ memory_percent‚99.96359348297119& proc.memory.availableMB‚ 5.96484375$ disk./.usageGB‚1166.394287109375# disk.nvme0n1.out‚73567.61328125 disk.sda3.in‚0 disk.sda3.out‚ 88.859375" proc.memory.rssMB‚ 3338.3515625* proc.memory.percent‚20.375680923461914 cpu‚25.123478201567448 proc.cpu.threads‚17 network.recv‚911285( disk./.usagePercent‚78.2776992779031" disk.nvme0c0n1.in‚ 6495.8671875 disk.sda.in‚0 disk.sda.out‚ 88.859375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.4296875 disk.sda2.out‚0 network.sent‚ 2.53269e+06% disk.nvme0c0n1.out‚73567.61328125 disk.sda1.in‚0 disk.dm-1.out‚ 88.859375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 6495.8671875 disk.dm-0.out‚73567.61328125 disk.dm-1.in‚0󉑓ý ‚(:ˆ ®´èÔ³‘òß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚45.0 gpu.0.powerWatts‚227.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.5242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ ¼“þ ‚(:ˆ ½´èÔ…ì³à gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚46.0 gpu.0.powerWatts‚247.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û5|pÿ ‚(:å ½´èÔçËéà disk.sda.out‚ 89.2421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.43359375 disk.sda2.in‚0$ proc.memory.availableMB‚5.953125 network.recv‚916182 disk.dm-0.in‚ 6495.8671875# disk.dm-0.out‚73688.43505859375" disk.nvme0c0n1.in‚ 6495.8671875( disk.nvme0c0n1.out‚73688.43505859375 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚99.96366500854492 network.sent‚ 2.538856e+06) disk./.usagePercent‚78.31139294149953 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 89.2421875 disk.sda2.out‚0" proc.memory.rssMB‚ 3338.3515625 cpu‚25.124239484637645% disk./.usageGB‚1166.8963470458984& disk.nvme0n1.out‚73688.43505859375* proc.memory.percent‚20.375680923461914 proc.cpu.threads‚17 disk.nvme0n1.in‚ 6495.8671875 disk.sda3.out‚ 89.2421875 disk.sda.in‚02Aãmt€ ‚(:é Ì´èÔêè¯Þ" proc.memory.rssMB‚ 3338.3515625 cpu‚25.12505672754593 proc.cpu.threads‚17 network.sent‚ 2.544686e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚73905.04833984375 disk.sda3.in‚0* proc.memory.percent‚20.375680923461914) disk./.usagePercent‚78.31125392940844% disk./.usageGB‚1166.8942756652832" disk.nvme0c0n1.in‚ 6495.8671875( disk.nvme0c0n1.out‚73905.04833984375 disk.sda.out‚ 89.70703125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 89.70703125 network.recv‚920587 disk.drbd1260.out‚ 0.43359375 disk.sda2.in‚0 disk.nvme0n1.in‚ 6495.8671875& disk.nvme0n1.out‚73905.04833984375 disk.sda3.out‚ 89.70703125$ memory_percent‚99.96521472930908& proc.memory.availableMB‚ 5.69921875 disk.dm-0.in‚ 6495.8671875äÊ.ž ‚(:“ Ì´èÔš­Øß gpu.0.gpu‚84.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚45.0 gpu.0.powerWatts‚248.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.693799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éØ’O‚ ‚(:Ä Û´èÔóݶÞ proc.cpu.threads‚17 disk.sda1.in‚0! disk.nvme0n1.out‚ 74218.046875# memory_percent‚99.9645471572876 cpu‚25.12598577679312 network.recv‚924080# disk.nvme0c0n1.out‚ 74218.046875 disk.sda.out‚90.375 disk.dm-1.in‚0 disk.dm-1.out‚90.375 disk.dm-0.out‚ 74218.046875 disk.sda3.in‚0& proc.memory.availableMB‚ 5.80859375% disk./.usageGB‚1167.3961486816406" disk.nvme0c0n1.in‚ 6640.1640625 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.4375 disk.sda2.out‚0 disk.nvme0n1.in‚ 6640.1640625 network.sent‚ 2.54992e+06( disk./.usagePercent‚78.3449350486357 disk.sda2.in‚0 disk.dm-0.in‚ 6640.1640625 disk.sda3.out‚90.375" proc.memory.rssMB‚ 3338.3515625* proc.memory.percent‚20.375680923461914ì/iÕ’ƒ ‚(:‡ Û´èÔ·§Æà gpu.0.gpu‚41.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.451' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò¡#\„ ‚(:Ñ ê´èÔ°Ææ# proc.memory.rssMB‚ 3338.35546875 proc.cpu.threads‚17 network.recv‚927137% disk./.usageGB‚1167.2707214355469 disk.nvme0n1.out‚ 74651.03125 disk.sda3.in‚0 disk.sda.out‚ 90.8359375 disk.sda1.in‚0 disk.drbd1260.out‚0.4375 disk.sda2.in‚0 network.sent‚ 2.554783e+06" disk.nvme0c0n1.in‚ 6640.1640625" disk.nvme0c0n1.out‚ 74651.03125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 90.8359375 disk.sda2.out‚0& proc.memory.availableMB‚ 6.05078125 disk.dm-0.out‚ 74651.03125 disk.sda3.out‚ 90.8359375 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚99.96306896209717* proc.memory.percent‚20.375704765319824 cpu‚25.12661612765024) disk./.usagePercent‚78.33651752090988 disk.dm-0.in‚ 6640.1640625 disk.nvme0n1.in‚ 6640.1640625·%Îœ… ‚(:‘ ê´èÔì³½é gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó-[r† ‚(:ç ù´èÔÚìµß disk.sda.out‚ 91.12109375 disk.sda1.out‚0 disk.dm-1.out‚ 91.12109375 disk.drbd1260.out‚ 0.44140625 disk.dm-0.in‚ 6640.1640625% disk.nvme0n1.out‚75116.6787109375 disk.sda3.out‚ 91.12109375) disk./.usagePercent‚78.34868479098972 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 6640.1640625$ memory_percent‚99.96457099914551% proc.memory.availableMB‚ 5.8046875# proc.memory.rssMB‚ 3338.35546875 network.sent‚ 2.561207e+06 disk.dm-1.in‚0 disk.sda2.in‚0" disk.dm-0.out‚75116.6787109375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 6640.1640625' disk.nvme0c0n1.out‚75116.6787109375 network.recv‚932007 disk.sda2.out‚0* proc.memory.percent‚20.375704765319824 cpu‚25.127402099337058 proc.cpu.threads‚17% disk./.usageGB‚1167.4520225524902Ô®ïÕž‡ ‚(:“ ù´èÔ”Ž«ß gpu.0.gpu‚94.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11286926269531* gpu.0.memoryAllocatedBytes‚ 58508509184 gpu.0.temp‚45.0 gpu.0.powerWatts‚252.206' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.441199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔìÊç2ˆ - _wandbruntime‚3056  _runtime‚30560!zsʼn ¿ & lr-AdamW/pg1‚0.0009969190932223398 & lr-AdamW/pg2‚0.0009969190932223398 trainer/global_step‚1049 $ _timestamp‚1.788484228915579e+09 _runtime‚3056.457370584 _step‚81Ã0!ÉŠ à trainer/global_step‚1049 $ _timestamp‚1.788484228915579e+09 _runtime‚3056.457370584 _step‚81 & lr-AdamW/pg1‚0.0009969190932223398 & lr-AdamW/pg2‚0.0009969190932223398Q ›S­2‹ - _wandbruntime‚3056  _runtime‚3056ÄÝ~ EŒ ìO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21751373 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5835104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 579547¤Ì±Q­&95520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21751379 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4072118125568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172698148245 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2173 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340126902906880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99089408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326150020311941 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2173 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336949863036416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3176264985577 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25868677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69133625085440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128187392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4557111296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57954795520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336949863036416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3180608699904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322977322163696 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14808410 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69133496898048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 _runtime‚3056.461549791 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47620056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5309 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340130497164288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336949888464384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4072217214976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5309 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25868677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3177039870464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25842595 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2173 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚829 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3180608699904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340130497164288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8974135 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚82 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1717394944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65061407870464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21751379 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884842289193926e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3177039870464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47593968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5835107 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326153612577241 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65061378772480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 ÑMYö' ðO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 785935872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4557111296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336949888464384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322977347591664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4557111296 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25842595 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340126902906880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21751373 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57954795520 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5309 trainer/global_step‚1049 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322977322163696 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14808410 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69133496898048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 _runtime‚3056.461549791 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47620056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5309 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340130497164288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336949888464384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4072217214976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5309 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25868677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3177039870464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25842595 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2173 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚829 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3180608699904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340130497164288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8974135 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚82 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1717394944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65061407870464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21751379 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884842289193926e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3177039870464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47593968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5835107 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326153612577241 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65061378772480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21751373 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5835104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57954795520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21751379 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4072118125568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172698148245 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2173 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340126902906880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99089408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326150020311941 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2173 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336949863036416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3176264985577 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25868677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69133625085440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128187392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4557111296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57954795520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336949863036416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3180608699904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337264479480832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3183603299840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69198850080768 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337264479480832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21771629 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326458274087263 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101173248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3175688660859 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14822116 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 785935872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4557111296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3180034457088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69198980352000 _runtime‚ 3058.97097976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1717394944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326454681821755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2173 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8982365 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 _step‚83 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2173 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47664624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3179255498399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25892995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21771623 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340448082780672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4075925432320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337264454052864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25866887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5840591 trainer/global_step‚1049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14822956 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323278993160896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8981528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25866887A÷zœ  úM % _timestamp‚1.7884842314286036e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47638510 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3183603299840 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14822116 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 785935872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4557111296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3180034457088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69198980352000 _runtime‚ 3058.97097976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1717394944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326454681821755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2173 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8982365 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 _step‚83 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2173 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47664624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3179255498399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25892995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.f¾öCôreed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21771623 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340448082780672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4075925432320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337264454052864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25866887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5840591 trainer/global_step‚1049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14822956 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323278993160896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8981528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25866887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3180034457088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337264479480832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3183603299840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69198850080768 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337264479480832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21771629 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326458274087263 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101173248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3175688660859 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352679439286784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3326797029888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68138701668864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚774 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6109304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72401878708736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338060872023696 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356006210888704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6109307 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3194.547985359 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22763879 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27060547 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352679413858816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22763879 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27060547 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3325817788285 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22763873 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27086629 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356009805146112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 _step‚86 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49850508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22763873 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15493874 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3322250950953 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356009805146112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341386715239949 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136ó[k« ðO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 VFDeviceStatsMoæÿ‹'nitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3326797029888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356006210888704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9385344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68138672570880 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352679439286784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3326797029888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68138701668864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚774 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6109304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72401878708736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018902528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338060872023696 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356006210888704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6109307 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3194.547985359 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22763879 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27060547 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352679413858816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22763879 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27060547 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3325817788285 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22763873 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27086629 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356009805146112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 _step‚86 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49850508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22763873 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15493874 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3322250950953 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356009805146112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341386715239949 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15494651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9384570 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72401756812800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338060897451664V†³…1“¬ ‚(:ˆ ¶èÔÁÅ¥â gpu.0.gpu‚96.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚49.0 gpu.0.powerWatts‚331.148' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÒÐÊr­ ‚(:ç ¶èÔîÞÙâ disk.sda.out‚ 95.58984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.4609375 network.sent‚ 2.697873e+06% disk./.usageGB‚1169.2915077209473& disk.nvme0n1.out‚78490.72021484375 disk.sda3.out‚ 95.58984375" disk.nvme0c0n1.in‚ 7077.9921875( disk.nvme0c0n1.out‚78490.72021484375 disk.sda.in‚0 disk.dm-1.out‚ 95.58984375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚78490.72021484375" memory_percent‚99.965500831604& proc.memory.availableMB‚ 5.65234375# proc.memory.rssMB‚ 3338.35546875 disk.dm-0.in‚ 7077.9921875* proc.memory.percent‚20.375704765319824 cpu‚25.13839976318804 proc.cpu.threads‚17 network.recv‚974575) disk./.usagePercent‚78.47213418407573 disk.nvme0n1.in‚ 7077.9921875 disk.sda3.in‚0aûÂ2® - _wandbruntime‚3197  _runtime‚3197­ ^;ü&¯ öM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27084839 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15507605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338362568448864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚87 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚784 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884843695353405e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22784123 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356327390762496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3328808301107 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49895076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6114788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72467134851072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4266916404224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356327390762496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356323796491776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27084839 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882624 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3329791616512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3325241463567 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 trainer/global_step‚1099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338362543020896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49868962 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352994030303232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68200218446848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27110947 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3333360459264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9392817 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49895076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6cûÚ'° úM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882624 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3329791616512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3325241463567 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 trainer/global_step‚1099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338362543020896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49868962 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352994030303232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68200218446848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27110947 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3333360459264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9392817 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49895076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27084839 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15507605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338362568448864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚87 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚784 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884843695353405e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22784123 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356327390762496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3328808301107 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49895076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6114788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72467134851072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4266916404224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018902528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356327390762496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356323796491776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27084839 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108Wù‹ð$2± - _wandbruntime‚3197  _runtime‚3197ÜîhÒز Ô /loss/FRt4/assignment_loss‚1.561383068561554 /loss/FBt3/detection_loss‚2.3402001351118087 _step‚88 epoch‚1 /loss/FRt3/assignment_loss‚4.518370146723999 2loss/SRqqt1/assignment_loss‚2.0989733769868812 2loss/FBt4/assignment_loss‚0.049902456155842226 .loss/FRt1/detection_loss‚3.209356811913577 /loss/FRt3/detection_loss‚2.5943397194489664 /loss/SRqqt2/detection_loss‚2.48112502330687 /loss/FRt1/assignment_loss‚4.517507466402921 0loss/FRt2/assignment_loss‚2.8303204611107544 2loss/SRqqt4/assignment_loss‚0.9613938934892104 /loss/FBt4/detection_loss‚2.1398048632674747 % _timestamp‚1.7884843695381963e+09 _runtime‚ 3197.07936024 /loss/FRt4/detection_loss‚1.8522167897354516 1loss/SRqqt1/detection_loss‚2.9429752410699925 trainer/global_step‚1099 2loss/SRqqt2/assignment_loss‚1.8351826818614472 1loss/FBt1/assignment_loss‚0.28023663079135175 1loss/FBt2/assignment_loss‚0.11570030288323563 1loss/FBt3/assignment_loss‚0.26665740019331374 /loss/FBt2/detection_loss‚1.0937682209855855 1loss/SRqqt3/detection_loss‚2.1193380024231656 /loss/FBt1/detection_loss‚2.8359078389626964 $loss/total_loss‚47.4192224364791 2loss/SRqqt3/assigì”ÿa‚nment_loss‚1.7436207414303027 .loss/FRt2/detection_loss‚2.011657745710441 1loss/SRqqt4/detection_loss‚1.0192834179522647¨ˆ¦^³ Ø /loss/FRt4/detection_loss‚1.8522167897354516 1loss/SRqqt1/detection_loss‚2.9429752410699925 trainer/global_step‚1099 2loss/SRqqt2/assignment_loss‚1.8351826818614472 1loss/FBt1/assignment_loss‚0.28023663079135175 1loss/FBt2/assignment_loss‚0.11570030288323563 1loss/FBt3/assignment_loss‚0.26665740019331374 /loss/FBt2/detection_loss‚1.0937682209855855 1loss/SRqqt3/detection_loss‚2.1193380024231656 /loss/FBt1/detection_loss‚2.8359078389626964 $loss/total_loss‚47.4192224364791 2loss/SRqqt3/assignment_loss‚1.7436207414303027 .loss/FRt2/detection_loss‚2.011657745710441 1loss/SRqqt4/detection_loss‚1.0192834179522647 /loss/FRt4/assignment_loss‚1.561383068561554 /loss/FBt3/detection_loss‚2.3402001351118087 _step‚88 epoch‚1 /loss/FRt3/assignment_loss‚4.518370146723999 2loss/SRqqt1/assignment_loss‚2.0989733769868812 2loss/FBt4/assignment_loss‚0.049902456155842226 .loss/FRt1/detection_loss‚3.209356811913577 /loss/FRt3/detection_loss‚2.5943397194489664 /loss/SRqqt2/detection_loss‚2.48112502330687 /loss/FRt1/assignment_loss‚4.517507466402921 0loss/FRt2/assignment_loss‚2.8303204611107544 2loss/SRqqt4/assignment_loss‚0.9613938934892104 /loss/FBt4/detection_loss‚2.1398048632674747 % _timestamp‚1.7884843695381963e+09 _runtime‚ 3197.07936024X íôq´ ‚(:æ ž¶èÔ¸‹†Þ disk.sda.out‚ 96.2890625 disk.sda1.out‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 5.5859375# proc.memory.rssMB‚ 3338.35546875* proc.memory.percent‚20.375704765319824% disk./.usageGB‚1169.2919540405273 disk.dm-0.in‚ 7077.9921875 disk.dm-1.in‚0 disk.dm-1.out‚ 96.2890625 cpu‚25.139364125733017 proc.cpu.threads‚17 network.sent‚ 2.772451e+06 disk.sda3.in‚0( disk.nvme0c0n1.out‚78873.38427734375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4609375 network.recv‚979017) disk./.usagePercent‚78.47216413695723 disk.nvme0n1.in‚ 7077.9921875& disk.nvme0n1.out‚78873.38427734375 disk.sda3.out‚ 96.2890625" disk.nvme0c0n1.in‚ 7077.9921875 disk.sda2.in‚0$ memory_percent‚99.96590614318848# disk.dm-0.out‚78873.38427734375 disk.sda.in‚0ƒÐÜñžµ ‚(:“ ž¶èÔ²²Þ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚251.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.397000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LtÄÃm¶ ‚(:â ­¶èÔƒ¤Áã disk.sda2.out‚0 disk.nvme0n1.in‚ 7077.9921875 disk.sda3.out‚ 96.66796875$ memory_percent‚99.96440410614014# proc.memory.rssMB‚ 3338.35546875 disk.dm-1.in‚0 disk.dm-1.out‚ 96.66796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.46484375 network.sent‚ 2.851197e+06 network.recv‚984771% disk.nvme0c0n1.out‚79279.10546875 disk.sda.out‚ 96.66796875% disk./.usageGB‚1169.2684860229492 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚79279.10546875 disk.sda3.in‚0& proc.memory.availableMB‚ 5.83203125* proc.memory.percent‚20.375704765319824) disk./.usagePercent‚78.47058917860659" disk.nvme0c0n1.in‚ 7077.9921875 disk.sda2.in‚0 disk.dm-0.in‚ 7077.9921875# disk.nvme0n1.out‚79279.10546875 cpu‚25.140461816051292 proc.cpu.threads‚17gv‘· ‚(:† ­¶èÔ§“Ôå gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚41.0 gpu.0.powerWatts‚115.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þq\X“¸ ‚(:ˆ ¼¶èÔ´ÐùÝ gpu.0.gpu‚68.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚43.0 gpu.0.powerWatts‚203.021' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.6042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>»dah¹ ‚(:Ý ¼¶èÔ°Ú˜Þ disk.drbd1260.in‚0 disk.dm-0.out‚79779.71484375% disk.nvme0c0n1.out‚79779.71484375 disk.sda.in‚0 disk.sda.out‚ 96.9765625 disk.sda1.out‚0$ memory_percent‚99.96614456176758$ proc.memory.availableMB‚5.546875 network.sent‚ 2.855252e+06% disk./.usageGB‚1168.7965812683105 disk.drbd1260.out‚ 0.46484375 disk.sda2.out‚0 disk.dm-0.in‚ 7077.9921875 disk.nvme0n1.in‚ 7077.9921875" disk.nvme0c0n1.in‚ 7077.9921875* proc.memory.percent‚20.375704765319824) disk./.usagePercent‚78.43891925456832 disk.dm-1.in‚0 disk.sda3.out‚ 96.9765625# proc.memory.rssMB‚ 3338.35546875 cpu‚25.141494130916673 proc.cpu.threads‚17 network.recv‚989050 disk.dm-1.out‚ 96.9765625 disk.sda2.in‚0# disk.nvme0n1.out‚79779.71484375 disk.sda3.in‚0 disk.sda1.in‚0€Ñ=rº ‚(:ç ˶èÔÅô¹Þ disk.sda2.out‚0( disk.nvme0c0n1.out‚79935.39208984375$ memory_percent‚99.96626377105713& proc.memory.availableMB‚ 5.52734375 network.sent‚ 2.860333e+06 network.recv‚992654 disk.drbd1260.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚79935.39208984375" disk.nvme0c0n1.in‚ 7104.5078125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 7104.7109375# disk.dm-0.out‚79935.39208984375 disk.sda3.in‚0 disk.sda3.out‚ 97.42578125 disk.dm-1.in‚0 disk.dm-1.out‚ 97.42578125* proc.memory.percent‚20.375704765319824 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7104.6328125 disk.sda.out‚ 97.42578125 disk.sda1.out‚0# proc.memory.rssMB‚ 3338.35546875 cpu‚25.142354096124002) disk./.usagePercent‚78.43891669449297$ disk./.usageGB‚1168.796543121338 disk.drbd1260.out‚0.46875+¼¾‘ž» ‚(:“ ˶èÔ™ ˜ß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚45.0 gpu.0.powerWatts‚252.155' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.431000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F i½l¼ ‚(:á Ú¶èÔܪÕã disk.dm-0.in‚ 7196.1015625 disk.sda3.out‚ 97.796875# proc.memory.rssMB‚ 3338.35546875% disk./.usageGB‚1168.7730751037598 disk.dm-1.in‚0& disk.nvme0n1.out‚80233.93505859375& proc.memory.availableMB‚ 5.50390625* proc.memory.percent‚20.375704765319824 network.recv‚996265 disk.sda.in‚0 disk.dm-1.out‚ 97.796875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.46875# disk.dm-0.out‚80233.93505859375 disk.nvme0n1.in‚ 7196.1015625 cpu‚25.14327822149083) disk./.usagePercent‚78.43734173614233 disk.sda.out‚ 97.796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚99.96640682220459 proc.cpu.threads‚17 network.sent‚ 2.867403e+06" disk.nvme0c0n1.in‚ 7196.1015625( disk.nvme0c0n1.out‚80233.93505859375 disk.sda2.in‚0eÁ “½ ‚(:ˆ Ú¶èÔÚ°Îå gpu.0.gpu‚92.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚231.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.3852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ£¯xx¾ ‚(:í é¶èÔ³Á½â disk.sda1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚20.375704765319824 network.sent‚ 2.871683e+06 disk.sda.out‚ 98.23828125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 98.23828125 disk.drbd1260.out‚ 0.47265625 disk.sda2.out‚0# proc.memory.rssMB‚ 3338.35546875 network.recv‚ 1.001054e+06) disk./.usagePercent‚78.43812819128752' disk.nvme0c0n1.out‚80541.4892578125 disk.sda2.in‚0" disk.dm-0.out‚80541.4892578125 disk.nvme0n1.in‚ 7196.1015625% disk.nvme0n1.out‚80541.4892578125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 7196.1015625 disk.sda.in‚0 disk.dm-0.in‚ 7196.1015625 disk.sda3.out‚ 98.23828125$ memory_percent‚99.96485710144043% proc.memory.availableMB‚ 5.7578125 cpu‚25.144293777719973 proc.cpu.threads‚17% disk./.usageGB‚1168.7847938537598Þ×G›¿ ‚(: é¶èÔ™†”á gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚45.0 gpu.0.powerWatts‚315.105' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚63.02100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aù¨¥“À ‚(:ˆ ø¶èÔÙûïÝ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚45.0 gpu.0.powerWatts‚245.296' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇG4|eÁ ‚(:Ú ø¶èÔýæ…ß" disk.nvme0c0n1.in‚ 7196.1015625 disk.sda1.in‚0 network.sent‚ 2.877565e+06 disk.dm-0.in‚ 7196.1015625 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.47265625& proc.memory.availableMB‚ 5.49609375# proc.memory.rssMB‚ 3338.35546875* proc.memory.percent‚20.375704765319824 disk.nvme0n1.in‚ 7196.1015625% disk.nvme0n1.out‚81128.4580078125 disk.sda.out‚99.25 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚99.96645450592041 proc.cpu.threads‚17 network.recv‚ 1.005594e+06% disk./.usageGB‚1168.7730751037598' disk.nvme0c0n1.out‚81128.4580078125 disk.dm-1.out‚99.25 disk.drbd1260.in‚0 disk.sda2.out‚0 cpu‚25.1452611204162) disk./.usagePercent‚78.43734173614233" disk.dm-0.out‚81128.4580078125 disk.sda3.out‚99.25(Ðáx ‚(:í ‡·èÔÏÖ›ß disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 99.9296875( disk.nvme0c0n1.out‚81450.15380859375# proc.memory.rssMB‚ 3338.35546875* proc.memory.percent‚20.375704765319824 cpu‚25.146385841837805 proc.cpu.threads‚17 disk.sda2.out‚0 network.recv‚ 1.009258e+06% disk./.usageGB‚1168.7730751037598 disk.sda1.out‚0 disk.dm-1.out‚ 99.9296875 disk.nvme0n1.in‚ 7196.1015625 disk.sda3.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚81450.15380859375" disk.nvme0c0n1.in‚ 7196.1015625$ memory_percent‚99.96724128723145% proc.memory.availableMB‚ 5.3671875 disk.dm-0.in‚ 7196.1015625# disk.dm-0.out‚81450.15380859375 network.sent‚ 2.882668e+06) disk./.usagePercent‚78.43734173614233 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.48046875 disk.sda.in‚0 disk.sda.out‚ 99.9296875vYt7’à ‚(:‡ ‡·èÔñ—à gpu.0.gpu‚71.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚247.175' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ãìÄ ‚(:ô –·èÔ›¼µÞ disk.sda1.out‚0 disk.drbd1260.out‚ 0.48046875 disk.sda2.in‚0& disk.nvme0n1.out‚81902.72412109375 disk.sda.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 5.37109375* proc.memory.percent‚20.375704765319824 cpu‚25.1473681901069 proc.cpu.threads‚17 network.recv‚ 1.012769e+06# disk.dm-0.out‚81902.72412109375 disk.dm-1.in‚0$ memory_percent‚99.96721744537354# proc.memory.rssMB‚ 3338.35546875 network.sent‚ 2.887757e+06 disk.sda3.in‚0 disk.sda3.out‚ 100.43359375# disk.nvme0c0n1.in‚ 7315.05859375 disk.sda.out‚ 100.43359375 disk.sda1.in‚0 disk.dm-1.out‚ 100.43359375 disk.drbd1260.in‚0) disk./.usagePercent‚78.43891669449297$ disk./.usageGB‚1168.796543121338 disk.dm-0.in‚ 7315.05859375! disk.nvme0n1.in‚ 7315.05859375( disk.nvme0c0n1.out‚81902.72412109375…çò’Å ‚(:‡ –·èÔͪŸß gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚229.785' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.957 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÊ2Æ - _wandbruntime‚3333  _runtime‚3333é¼tlÆÇ À _runtime‚3333.697665007 _step‚89 & lr-AdamW/pg1‚0.0009886553543850434 & lr-AdamW/pg2‚0.0009886553543850434 trainer/global_step‚1149 % _timestamp‚1.7884845061560829e+09Ö…úëÊÈ Ä & lr-AdamW/pg1‚0.0009886553543850434 & lr-AdamW/pg2‚0.0009886553543850434 trainer/global_step‚1149 % _timestamp‚1.7884845061560829e+09 _runtime‚3333.697665007 _step‚89Y9-G§2É - _wandbruntime‚3333  _runtime‚3333_×w=ò'Ê ìO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28278499 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6383507 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353144447311664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353144421883696 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 % _timestamp‚1.7884845061585507e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126090240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71215995467264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368408990109184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3476554189312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1149 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4455407561728 _step‚90 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 _runtime‚3333.700453032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371885518870528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368408990109184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96992256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368408964681216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3480123018752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52080960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4455310569472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23776373 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75671403028992 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018902528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16181692 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23776373 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371889113127936 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚803 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71215966369280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356619817902657 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23776379 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23776379 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6383504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837332 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚803 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71215966369280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356619817902657 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23776379 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23776379 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6383504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353144447311664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353144421883696 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 % _timestamp‚1.7884845061585507e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126090240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71215995467264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368408990109184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3476554189312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1149 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4455407561728 _step‚90 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 _runtime‚3333.700453032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371885518870528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 TDDeviceStatsMonitor.on_train_batch_sµhÿø¯tart/active_bytes.large_pool.peak‚ 52238108672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368408990109184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96992256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368408964681216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3480123018752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52080960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4455310569472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23776373 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680Z5 #2Ì - _wandbruntime‚3336  _runtime‚3336UÒ\ù&Í óM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3478361103815 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _step‚91 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23796629 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75736781728768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23796623 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52125528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16194507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16195293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71277541343232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368723555697664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3474794266275 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28328899 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372206698744320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353446118308864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28328899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356920887147171 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75736657748992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3483117618688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 # _timestamp‚1.78848450886098e+09 _runtime‚ 3336.40328541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52125528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚368723581125632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3483117618688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4459240385536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372206698744320 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚786 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9805519 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4459145503744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6388991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚368723581125632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3479548775936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9806302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚783 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356924479412679 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3479548775936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353446092880896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23796623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71277512245248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23796629 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28302791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174¯Òoàý&Î ÷M WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚786 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9805519 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4459145503744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6388991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚368723581125632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3479548775936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9806302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚783 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356924479412679 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3479548775936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353446092880896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23796623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71277512245248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23796629 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28302791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3478361103815 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _step‚91 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23796629 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75736781728768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23796623 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52125528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16194507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16195293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71277541343232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368723555697664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3474794266275 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28328899 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372206698744320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353446118308864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28328899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356920887147171 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75736657748992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3483117618688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 # _timestamp‚1.78848450886098e+09 _runtime‚ 3336.40328541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52125528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚368723581125632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3483117618688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4459240385536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372206698744320[Aç•i2Ï - _wandbruntime‚3336  _runtime‚3336<ê,XÐ Ò 1loss/SRqqt2/detection_loss‚2.3834825575351717 0loss/FBt1/assignment_loss‚0.3192942728638328 /loss/FRt1/detection_loss‚3.2268566566964854 trainer/global_step‚1149 % _timestamp‚1.7884845088651454e+09 _runtime‚3336.406738539 /loss/FRt3/assignment_loss‚4.426966376793691 2loss/SRqqt4/assignment_loss‚0.9121764458682202 0loss/SRqqt1/detection_loss‚2.548203098773956 %loss/total_loss‚46.96598970999873 1loss/SRqqt1/assignment_loss‚2.050195848817627 1loss/FBt3/assignment_loss‚0.21983957745773108 .loss/FBt2/detection_loss‚0.966404267307371 /loss/FBt4/detection_loss‚2.6730541363358498 epoch‚1 /loss/FRt2/assignment_loss‚2.843406245913557 2loss/SRqqt2/assignment_loss‚1.7002893725203143 2loss/SRqqt3/assignment_loss‚1.5179917576339315 /loss/FRt3/detection_loss‚2.4236621253001385 .loss/FBt1/detection_loss‚2.207454336852562 0loss/FRt1/assignment_loss‚4.3906828009563945 .loss/FRt2/detection_loss‚2.022329998803475 0loss/FRt4/assignment_loss‚1.9095200739123606 1loss/FBt2/assignment_loss‚0.23731314975679763 0loss/FBt4/assignment_loss‚0.0869387624893534 1loss/SRqqt4/detection_loss‚1.1334238735225675 /loss/FBt3/detection_loss‚2.7045014932237823 _step‚92 1loss/SRqqt3/detection_loss‚2.0202287502148573 .loss/FRt4/detection_loss‚2.041773730448701ÃjƒÅ\Ñ Ö .loss/FRt4/detection_loss‚2.041773730448701 1loss/SRqqt2/detection_loss‚2.3834825575351717 0loss/FBt1/assignment_loss‚0.3192942728638328 /loss/FRt1/detection_loss‚3.2268566566964854 trainer/global_step‚1149 % _timestamp‚1.7884845088651454e+09 _runtime‚3336.406738539 /loss/FRt3/assignment_loss‚4.426966376793691 2loss/SRqqt4/assignment_loss‚0.9121764458682202 0loss/SRqqt1/detection_loss‚2.548203098773956 %loss/total_loss‚46.96598970999873 1loss/SRqqt1/assignment_loss‚2.050195848817627 1loss/FBt3/assignment_loss‚0.21983957745773108 .loss/FBt2/detection_loss‚0.966404267307371 /loss/FBt4/detection_loss‚2.6730541363358498 epoch‚1 /loss/FRt2/assignment_loss‚2.843406245913557 2loss/SRqqt2/assignment_loss‚1.7002893725203143 2loss/SRqqt3/assignment_loss‚1.5179917576339315 /loss/FRt3/detection_loss‚2.4236621253001385 .loss/FBt1/detection_loss‚2.207454336852562 0loss/FRt1/assignment_loss‚4.3906828009563945 .loss/FRt2/detection_loss‚2.022329998803475 0loss/FRt4/assignment_loss‚1.9095200739123606 1loss/FBt2/assignment_loss‚0.23731314975679763 0loss/FBt4/assignment_loss‚0.0869387624893534 1loss/SRqqt4/detection_loss‚1.1334238735225675 /loss/FBt3/detection_loss‚2.7045014932237823 _step‚92 1loss/SRqqt3/detection_loss‚2.0202287502148573\ZZeÿxÒ ‚(:í ¥·èÔëý³à proc.cpu.threads‚17 disk.sda3.out‚ 101.1171875) disk./.usagePercent‚78.43891669449297$ disk./.usageGB‚1168.796543121338 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚82153.4033203125& proc.memory.availableMB‚ 5.58984375 cpu‚25.14837231638466 disk.drbd1260.in‚0 disk.drbd1260.out‚0.484375' disk.nvme0c0n1.out‚82153.4033203125 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3338.35546875 network.recv‚ 1.017518e+06! disk.nvme0n1.in‚ 7315.05859375 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 101.1171875$ memory_percent‚99.96588230133057 network.sent‚ 2.893904e+06 disk.dm-0.in‚ 7315.05859375% disk.nvme0n1.out‚82153.4033203125# disk.nvme0c0n1.in‚ 7315.05859375 disk.sda.out‚ 101.1171875* proc.memory.percent‚20.375704765319824ìs•’“Ó ‚(:ˆ ¥·èÔ‹ªðà gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚332.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¾Ê|Ô ‚(:ñ ´·èÔ…µ¢ã disk.sda1.in‚0 disk.dm-0.in‚ 7315.05859375" disk.dm-0.out‚82364.5869140625% proc.memory.availableMB‚ 5.3359375# proc.memory.rssMB‚ 3338.35546875 network.recv‚ 1.023848e+06) disk./.usagePercent‚78.43891695050051% disk./.usageGB‚1168.7965469360352 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.375704765319824 cpu‚25.149063484461042' disk.nvme0c0n1.out‚82364.5869140625 disk.dm-1.out‚ 101.48046875 disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 101.48046875 disk.sda.out‚ 101.48046875 disk.sda1.out‚0 disk.drbd1260.out‚0.484375 disk.sda2.in‚0! disk.nvme0n1.in‚ 7315.05859375% disk.nvme0n1.out‚82364.5869140625$ memory_percent‚99.96743202209473 network.sent‚ 2.983076e+06# disk.nvme0c0n1.in‚ 7315.05859375 disk.sda.in‚0ô;"n“Õ ‚(:ˆ ´·èÔÞ¯å gpu.0.gpu‚89.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚324.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.8952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vœ·énÖ ‚(:ã ÷èÔ¡‚ÿè# proc.memory.rssMB‚ 3338.35546875) disk./.usagePercent‚78.36767235778925 disk.sda2.out‚0 disk.sda.in‚0$ memory_percent‚99.96590614318848 cpu‚25.149632804773038 proc.cpu.threads‚17 disk.sda3.out‚ 101.85546875 network.sent‚ 2.986262e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.48828125 disk.dm-0.in‚ 7315.05859375! disk.nvme0n1.in‚ 7315.05859375 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚ 1.027667e+06 disk.dm-0.out‚ 82736.78125 disk.nvme0n1.out‚ 82736.78125 disk.sda.out‚ 101.85546875 disk.sda1.in‚0 disk.dm-1.out‚ 101.85546875* proc.memory.percent‚20.375704765319824$ disk./.usageGB‚1167.734951019287 disk.sda2.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7315.05859375" disk.nvme0c0n1.out‚ 82736.78125% proc.memory.availableMB‚ 5.5859375§ðÌb’× ‚(:‡ ÷èÔÈ«ªé gpu.0.gpu‚24.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚312.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚62.5884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ä§goØ ‚(:ä Ò·èÔ£ÄÏá% proc.memory.availableMB‚ 5.5859375* proc.memory.percent‚20.375704765319824 cpu‚25.150250881502956! disk.nvme0n1.in‚ 7465.64453125 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 83218.9765625 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3338.35546875 network.sent‚ 2.99137e+06( disk./.usagePercent‚78.3660674465571$ disk./.usageGB‚1167.711036682129 disk.dm-0.out‚ 83218.9765625 disk.sda.out‚ 102.2890625 disk.drbd1260.out‚ 0.48828125$ memory_percent‚99.96590614318848 proc.cpu.threads‚17 network.recv‚ 1.031173e+06 disk.dm-0.in‚ 7465.64453125" disk.nvme0n1.out‚ 83218.9765625# disk.nvme0c0n1.in‚ 7465.64453125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 102.2890625 disk.sda.in‚0 disk.dm-1.out‚ 102.2890625fí«Ù ‚(:’ Ò·èÔ÷›€á gpu.0.gpu‚75.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚43.0 gpu.0.powerWatts‚206.723' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚41.34460000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kTÁxÚ ‚(:í á·èÔƒÁªè# disk.nvme0c0n1.in‚ 7465.64453125( disk.nvme0c0n1.out‚83506.71728515625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚83506.71728515625 network.sent‚ 2.999521e+06% disk./.usageGB‚1167.7341842651367 disk.sda2.out‚0! disk.nvme0n1.in‚ 7465.64453125# memory_percent‚99.9659538269043$ proc.memory.availableMB‚5.578125* proc.memory.percent‚20.375704765319824 cpu‚25.150723688338253 network.recv‚ 1.036096e+06 disk.sda.out‚ 102.734375 disk.dm-0.in‚ 7465.64453125& disk.nvme0n1.out‚83506.71728515625 disk.sda3.in‚0# proc.memory.rssMB‚ 3338.35546875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 102.734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4921875 disk.sda3.out‚ 102.734375) disk./.usagePercent‚78.36762090027487lhÜD“Û ‚(:ˆ á·èÔ¦µÈë gpu.0.gpu‚92.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚254.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.8756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–„ovÜ ‚(:ë ð·èÔÈçùã network.sent‚ 3.003506e+06 network.recv‚ 1.040248e+06 disk.sda3.out‚ 103.015625 disk.sda1.in‚0 cpu‚25.15135795358782 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 103.015625 disk.dm-1.out‚ 103.015625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7465.64453125$ memory_percent‚99.96757507324219& disk.nvme0n1.out‚83891.76318359375# disk.nvme0c0n1.in‚ 7465.64453125( disk.nvme0c0n1.out‚83891.76318359375 disk.dm-1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚83891.76318359375# proc.memory.rssMB‚ 3338.35546875 proc.cpu.threads‚17) disk./.usagePercent‚78.36761603613172% disk./.usageGB‚1167.7341117858887! disk.nvme0n1.in‚ 7465.64453125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.4921875" proc.memory.availableMB‚5.3125* proc.memory.percent‚20.375704765319824£°žÝ ‚(:“ ð·èÔÂÞéå gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.099' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.819799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÆr…rÞ ‚(:é ÿ·èÔðøßâ cpu‚25.15208248633244 proc.cpu.threads‚17 network.sent‚ 3.010522e+06! disk.nvme0n1.in‚ 7465.64453125 disk.dm-1.in‚0 disk.dm-1.out‚ 103.44140625 disk.sda2.out‚0# proc.memory.rssMB‚ 3338.35546875 disk.sda.out‚ 103.44140625 network.recv‚ 1.043819e+06% disk./.usageGB‚1167.7341117858887 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 84132.9921875 disk.drbd1260.in‚0 disk.dm-0.in‚ 7465.64453125" disk.nvme0n1.out‚ 84132.9921875# disk.nvme0c0n1.in‚ 7465.64453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.49609375 disk.s—ˆ7da2.in‚0% proc.memory.availableMB‚ 5.5703125) disk./.usagePercent‚78.36761603613172 disk.dm-0.out‚ 84132.9921875 disk.sda3.out‚ 103.44140625 disk.sda.in‚0$ memory_percent‚99.96600151062012* proc.memory.percent‚20.375704765319824Gy"ì“ß ‚(:ˆ ÿ·èÔ½ùËä gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚250.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.0088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ!u³và ‚(:ë Ž¸èÔß¿Òã network.recv‚ 1.047236e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 103.88671875 cpu‚25.152485818213037 network.sent‚ 3.013719e+06% disk./.usageGB‚1167.6115531921387# disk.nvme0c0n1.in‚ 7563.19921875$ disk.nvme0c0n1.out‚ 84229.6484375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚99.96774196624756" disk.nvme0n1.out‚ 84229.6484375 disk.sda3.out‚ 103.88671875) disk./.usagePercent‚78.35939102607165 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.49609375 disk.dm-0.in‚ 7563.19921875 disk.dm-0.out‚ 84229.6484375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 103.88671875& proc.memory.availableMB‚ 5.28515625# proc.memory.rssMB‚ 3338.35546875* proc.memory.percent‚20.375704765319824 proc.cpu.threads‚17! disk.nvme0n1.in‚ 7563.19921875EÑkëá ‚(:’ Ž¸èÔ»÷éã gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚254.092' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.81840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ÎU2â - _wandbruntime‚3460  _runtime‚34602\‡óÄã ¾ trainer/global_step‚1199 % _timestamp‚1.7884846328223894e+09 _runtime‚ 3460.3643764 _step‚93 & lr-AdamW/pg1‚0.0009839181928020205 & lr-AdamW/pg2‚0.0009839181928020205¸áÈä  & lr-AdamW/pg1‚0.0009839181928020205 & lr-AdamW/pg2‚0.0009839181928020205 trainer/global_step‚1199 % _timestamp‚1.7884846328223894e+09 _runtime‚ 3460.3643764 _step‚93]Æ?2å - _wandbruntime‚3460  _runtime‚3460 ,Ɉô'æ îO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3629880178176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3621356556369 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74293289265664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚368227997171664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384138515503616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24788873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚814 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54285324 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29522533 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384138540931584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29496451 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4647533976064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24788873 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54311412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚387768421109760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24788879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4647630968320 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96992256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384138515503616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3629880178176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3624923393701 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6657704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚387764826852352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74293260167680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚371852920565365 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 _step‚94 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚387764826852352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6657707 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368227971743696 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96992256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384138515503616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3629880178176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3624923393701 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6657704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚387764826852352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74293260167680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚371852920565365 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 _step‚94 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257408 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚387764826852352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457465856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6657707 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368227971743696 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3621356556369 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74293289265664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚368227997171664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384138515503616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24788873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚814 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54285324 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29522533 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384138540931584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29496451 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4647533976064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24788873 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54311412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚387768421109760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457465856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24788879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4647630968320 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16882142 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372153989809879 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29546851 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6663191 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29520743 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3632874778112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24809123 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 % _timestamp‚1.7884846351185553e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54355980 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54329866 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74354835141632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74354806043648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3624347068983 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29520743 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368529642740896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24809129 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54355980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚810 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6663188 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3627913906523 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384453131948032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96978944 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126076928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3629305935360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3632874778112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 _runtime‚3462.660585227 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54329866 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24809123 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24809129 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 VGDevice?T StatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 trainer/global_step‚1199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388082412455424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6³¡¯.'ê üM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10219764 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4651368948224 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74354835141632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74354806043648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3624347068983 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29520743 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368529642740896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24809129 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54355980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚810 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6663188 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3627913906523 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384453131948032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96978944 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594270720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126076928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265508 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018902528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3629305935360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3632874778112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 _runtime‚3462.660585227 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594270720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54329866 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24809123 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24809129 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837540 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 trainer/global_step‚1199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388082412455424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388086006726144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568842752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚95 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3629305935360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372157582075387 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16882955 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79006174991872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388086006726144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384453106520064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚813 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16882142 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372153989809879 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29546851 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6663191 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457465856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29520743 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3632874778112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457465856 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568842752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24809123 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 % _timestamp‚1.7884846351185553e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54355980 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54329866 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22201721370220184 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.33478349447250366 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46666666865348816 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5545522570610046 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2582891881465912 *jet/accuracy_1_of_4‚0.8646277785301208 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4197530746459961 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45359310507774353 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1423221379518509 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_6‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3638196587562561 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3019859194755554 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011554423719644547 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2242226004600525 -"Purity/1FRt*SRqqt3FBt/event_purity‚0.0625 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3192954659461975 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900455474853516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.989622175693512 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7938596606254578 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22440476715564728 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24152150750160217 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0947832390666008 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2131301611661911 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938082098960876 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5345940589904785 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41137006878852844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906367063522339 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3095238208770752 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33364659547805786 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909325838088989 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9953917264938354 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2547619044780731 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_1_of_2‚0.6772863864898682 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31606873869895935 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9815475344657898 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3962480127811432 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.38052448630332947 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2461896687746048 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19236606359481812 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_6‚0 trainer/global_step‚1207 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 _runtime‚3549.323792152 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.21467383205890656 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5525529384613037 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5041475296020508 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4073254466056824 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2086355835199356 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6401882767677307 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2380952537059784 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25270703434944153 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.09464286267757416 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3881487548351288 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 &particle/f_score‚0.383127897977829 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6058333516120911 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2473399043083191 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558467626571655 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.32760748267173767 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1599999964237213 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36657530069351196 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5833333134651184 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3456280529499054 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009677455760538578 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.21638092398643494 *jet/accuracy_1_of_3‚0.8157109022140503 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977303743362427 5 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.94047611951828 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚ 0.2265625 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5317592620849609 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4642857015132904 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18087559938430786 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8965518474578857 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41887757182121277 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20071429014205933 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.32499998807907104 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20171169936656952 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.47058823704719543 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004467751830816269 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.30917924642562866 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903247952461243 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 9"Purity/2FRt*SRqqt0/ò˜e2FBt/SRqqt_purity‚0.7763158082962036 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9753703474998474 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.002016128972172737 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.18630953133106232 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9786038398742676 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982544779777527 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9736841917037964 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21905000507831573 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.15000000596046448 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23287692666053772 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3855625092983246 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850910902023315 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990823864936829 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2575823664665222 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.322620689868927 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.30615076422691345 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0483240932226181 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.21923209726810455 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42195773124694824 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _step‚97 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.024378353729844093 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3074314594268799 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_6‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16428899765014648 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.384269654750824 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3089313209056854 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907549619674683 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3611111342906952 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20627276599407196 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013095238246023655 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30342918634414673 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9649122357368469 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12448975443840027 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41261616349220276 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9884268641471863 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3095238208770752 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012149142101407051 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3047832250595093 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23425482213497162 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015001018531620502 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2661663889884949 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3505062758922577 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9976660013198853 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4305555820465088 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8999999761581421 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚49.84353924239514 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.07067134976387024 particle/accuracy_1_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4967656433582306 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2888888716697693 8 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.34615379571914673 )jet/accuracy_2_of_5‚0.420947402715683 +particle/specificity‚0.9222281575202942 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0071428571827709675 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1676652729511261 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5408346056938171 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917038679122925 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2335381656885147 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3749999701976776 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37960582971572876 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9953917264938354 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44583335518836975 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6189626455307007 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34097543358802795 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0038416297174990177 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30868270993232727 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5782737731933594 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23350821435451508 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9927419424057007 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13849954307079315 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6428571343421936 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24921470880508423 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.331096351146698 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3888888657093048 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.572145402431488 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3684510290622711 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3166666626930237 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9898967742919922 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011627479456365108 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.26365846395492554 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19341562688350677 % _timestamp‚1.7884847217753978e+09 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.807964563369751 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5143041014671326 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15032680332660675 jet/accuracy_2_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19654446840286255 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40194445848464966 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.31285712122917175 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_3_of_7‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37719300389289856 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.349773645401001 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2233358919620514 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.69047611951828 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17197033762931824 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956106543540955 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 jet/accuracy_1_of_7‚1 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.98744136095047 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5459672212600708 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.00358422938734293 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2551221251487732 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.2391304075717926 +jet/accuracy_2_of_2‚0.10290972143411636 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38095760345458984 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16215313971042633 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14814814925193787 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2466084361076355 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3803921341896057 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34182485938072205 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.005376344081014395 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 +jet/accuracy_1_of_1‚0.39642414450645447 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 jet/accuracy_4_of_6‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.24432985484600067 >&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.27144965529441833 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9884414076805115 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2471652626991272 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9936347007751465 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22813181579113007 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6118665933609009 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.23529864847660065 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.46666666865348816 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38570576906204224 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2007831186056137 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012083047069609165 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.900793731212616 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21708805859088898 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.008064515888690948 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3888888657093048 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.26445579528808594 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009764555841684341 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0027649770490825176 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚ 0.2265625 jet/accuracy_6_of_6‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09221708029508591 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.40398547053337097 *jet/accuracy_1_of_5‚0.9203378558158875 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03377017006278038 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.29377880692481995 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.19675326347351074 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11666667461395264 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3089313209056854 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9827777743339539 particle/accuracy_2_of_6‚1 9"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4333333373069763 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9846453666687012 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.37107837200164795 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22979798913002014 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.028888756409287453 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3589743375778198 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.564573347568512 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02500000037252903 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.30775851011276245 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 5 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19746033847332 jet/accuracy_5_of_5‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5720751881599426 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973071217536926 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9900153875350952 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9479166865348816 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16428899765014648 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9967990517616272 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0021505376789718866 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869768023490906 particle/accuracy_4_of_4‚1 particle/accuracy_2_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6953388452529907 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.46875 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5192307829856873 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8307448625564575 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4513889253139496 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16458290815353394 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5942614674568176 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0071428571827709675 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3667507767677307 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9956210851669312 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.33772388100624084 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2222222238779068 particle/accuracy_7_of_7‚1 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.355890691280365 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,particle/sensitivity‚0.33657902479171753 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24660365283489227 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8868314027786255 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3076665997505188 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9851189851760864 jet/accuracy_4_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07320133596658707 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2380952537059784 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2651931345462799 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16669948399066925 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979222416877747 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883130788803101 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6709677577018738 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.091379314661026 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5348547101020813 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9706635475158691 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 epoch‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09991975128650665 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9022223949432373 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5890859961509705 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2111111283302307 9"Purity/1FRt0SRqqt3FBt/event_purity‚0.0714285746216774 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 +jet/accuracy_3_of_3‚0.02057380974292755 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.017777778208255768 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2101522982120514 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.24421298503875732 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_5_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49561405181884766 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1645941138267517 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 *"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0d´Û2Zü •§ @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.355890691280365 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,particle/sensitivity‚0.33657902479171753 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24660365283489227 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8868314027786255 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3076665997505188 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9851189851760864 jet/accuracy_4_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07320133596658707 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2380952537059784 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2651931345462799 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16669948399066925 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979222416877747 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883130788803101 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6709677577018738 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.091379314661026 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5348547101020813 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9706635475158691 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 epoch‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09991975128650665 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9022223949432373 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5890859961509705 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2111111283302307 9"Purity/1FRt0SRqqt3FBt/event_purity‚0.0714285746216774 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 +jet/accuracy_3_of_3‚0.02057380974292755 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.017777778208255768 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2101522982120514 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.24421298503875732 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_5_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49561405181884766 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1645941138267517 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 *"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982544779777527 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3717948794364929 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003150201519019902 +jet/accuracy_3_of_4‚0.06239637732505798 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12595276534557343 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.542616069316864 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt1SRqqt2FBt/event_proportion‚0.008757560513913631 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.31027716398239136 >&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22201721370220184 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.33478349447250366 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46666666865348816 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5545522570610046 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2582891881465912 *jet/accuracy_1_of_4‚0.8646277785301208 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4197530746459961 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45359310507774353 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1423221379518509 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_6‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3638196587562561 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3019859194755554 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011554423719644547 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2242226004600525 -"Purity/1FRt*SRqqt3FBt/event_purity‚0.0625 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3192954659461975 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900455474853516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_propËÇDByortion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.989622175693512 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7938596606254578 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22440476715564728 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24152150750160217 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0947832390666008 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2131301611661911 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938082098960876 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5345940589904785 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41137006878852844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906367063522339 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3095238208770752 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33364659547805786 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909325838088989 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9953917264938354 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2547619044780731 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_1_of_2‚0.6772863864898682 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31606873869895935 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9815475344657898 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3962480127811432 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.38052448630332947 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2461896687746048 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19236606359481812 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_6‚0 trainer/global_step‚1207 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 _runtime‚3549.323792152 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.21467383205890656 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5525529384613037 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5041475296020508 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4073254466056824 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2086355835199356 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6401882767677307 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2380952537059784 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25270703434944153 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.09464286267757416 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3881487548351288 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 &particle/f_score‚0.383127897977829 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6058333516120911 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2473399043083191 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558467626571655 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.32760748267173767 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1599999964237213 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36657530069351196 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5833333134651184 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3456280529499054 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009677455760538578 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.21638092398643494 *jet/accuracy_1_of_3‚0.8157109022140503 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977303743362427 5 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.94047611951828 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚ 0.2265625 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5317592620849609 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4642857015132904 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18087559938430786 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8965518474578857 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41887757182121277 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20071429014205933 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.32499998807907104 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20171169936656952 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.47058823704719543 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004467751830816269 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.30917924642562866 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903247952461243 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7763158082962036 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9753703474998474 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.002016128972172737 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.18630953133106232 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9786038398742676 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982544779777527 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9736841917037964 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21905000507831573 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.15000000596046448 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23287692666053772 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3855625092983246 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850910902023315 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990823864936829 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2575823664665222 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.322620689868927 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.30615076422691345 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0483240932226181 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.21923209726810455 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42195773124694824 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _step‚97 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.024378353729844093 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3074314594268799 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_6‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16428899765014648 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.384269654750824 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3089313209056854 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907549619674683 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3611111342906952 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20627276599407196 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013095238246023655 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30342918634414673 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9649122357368469 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12448975443840027 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41261616349220276 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9884268641471863 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3095238208770752 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012149142101407051 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3047832250595093 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23425482213497162 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015001018531620502 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2661663889884949 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3505062758922577 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9976660013198853 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4305555820465088 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8999999761581421 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚49.84353924239514 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.07067134976387024 particle/accuracy_1_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4967656433582306 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2888888716697693 8 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.34615379571914673 )jet/accuracy_2_of_5‚0.420947402715683 +particle/specificity‚0.9222281575202942 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0071428571827709675 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1676652729511261 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5408346056938171 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917038679122925 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2335381656885147 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3749999701976776 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37960582971572876 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9953917264938354 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44583335518836975 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6189626455307007 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34097543358802795 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0038416297174990177 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30868270993232727 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5782737731933594 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23350821435451508 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9927419424057007 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13849954307079315 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6428571343421936 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24921470880508423 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.331096351146698 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3888888657093048 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.572145402431488 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3684510290622711 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3166666626930237 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9898967742919922 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011627479456365108 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.26365846395492554 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19341562688350677 % _timestamp‚1.7884847217753978e+09 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.807964563369751 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5143041014671326 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15032680332660675 jet/accuracy_2_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19654446840286255 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40194445848464966 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.31285712122917175 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_3_of_7‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37719300389289856 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.349773645401001 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2233358919620514 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.69047611951828 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17197033762931824 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956106543540955 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 jet/accuracy_1_of_7‚1 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.98744136095047 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5459672212600708 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.00358422938734293 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2551221251487732 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.2391304075717926 +jet/accuracy_2_of_2‚0.10290972143411636 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38095760345458984 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16215313971042633 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14814814925193787 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2466084361076355 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3803921341896057 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34182485938072205 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.005376344081014395 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 +jet/accuracy_1_of_1‚0.39642414450645447 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 jet/accuracy_4_of_6‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.24432985484600067 >&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.27144965529441833 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9884414076805115 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2471652626991272 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9936347007751465 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22813181579113007 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6118665933609009 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.23529864847660065 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.46666666865348816 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38570576906204224 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2007831186056137 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012083047069609165 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.900793731212616 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21708805859088898 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.008064515888690948 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3888888657093048 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.26445579528808594 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009764555841684341 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0027649770490825176 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚ 0.2265625 jet/accuracy_6_of_6‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09221708029508591 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.40398547053337097 *jet/accuracy_1_of_5‚0.9203378558158875 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03377017006278038 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.29377880692481995 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.19675326347351074 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11666667461395264 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3089313209056854 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9827777743339539 particle/accuracy_2_of_6‚1 9"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4333333373069763 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9846453666687012 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.37107837200164795 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22979798913002014 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.028888756409287453 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3589743375778198 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.564573347568512 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02500000037252903 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.30775851011276245 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 5 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19746033847332 jet/accuracy_5_of_5‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5720751881599426 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973071217536926 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9900153875350952 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9479166865348816 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16428899765014648 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9967990517616272 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0021505376789718866 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869768023490906 particle/accuracy_4_of_4‚1 particle/accuracy_2_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6953388452529907 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.46875 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5192307829856873 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8307448625564575 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4513889253139496 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16458290815353394 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5942614674568176 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0071428571827709675 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3667507767677307 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9956210851669312 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.33772388100624084 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2222222238779068 particle/accuracy_7_of_7‚1 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052ažS÷@ý ‚(:… ÷¸èÔƒ…ÜÞ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚34.0 gpu.0.powerWatts‚93.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6554 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈXL yþ ‚(:î ÷¸èÔÜÙÐà disk.sda3.out‚ 107.32421875' disk.nvme0c0n1.out‚86728.1298828125 disk.sda.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 5837.79296875 cpu‚25.2467320990703$ disk./.usageGB‚1168.280689239502! disk.nvme0n1.in‚ 7698.60546875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7698.60546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0) proc.memory.percent‚35.63106060028076 network.sent‚ 3.137602e+06 network.recv‚ 1.084297e+06" disk.dm-0.out‚86728.1298828125% disk.nvme0n1.out‚86728.1298828125 disk.drbd1260.out‚0.515625$ memory_percent‚99.97556209564209& proc.memory.availableMB‚ 4.00390625 disk.dm-0.in‚ 7698.60546875 disk.dm-1.in‚0 disk.dm-1.out‚ 107.32421875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 107.32421875) disk./.usagePercent‚78.40429730765274}¥Þ"™ÿ ‚(:Ü †¹èÔ¬×ýÞ' disk.nvme0c0n1.out‚94290.9248046875 disk.sda.in‚0* proc.memory.percent‚20.412158966064453) disk./.usagePercent‚78.67266821374409 disk.nvme0n1.in‚ 7812.8984375 disk.sda3.in‚0 disk.sda.out‚107.625 disk.sda1.in‚0 disk.dm-1.out‚107.625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.515625 disk.dm-0.in‚ 7812.8984375" disk.nvme0c0n1.in‚ïù 8Î 7812.8984375 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚68.64852905273438$ proc.memory.availableMB‚5136.625! proc.memory.rssMB‚ 3344.328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚94290.9248046875 cpu‚25.241900031820233 network.sent‚ 3.242103e+06 network.recv‚ 1.090103e+06% disk./.usageGB‚1172.2796096801758" disk.dm-0.out‚94290.9248046875 disk.sda3.out‚107.625U³Š“€‚(:ˆ †¹èÔÃîúß gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚41.0 gpu.0.powerWatts‚231.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.2212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ @“‚(:ˆ •¹èÔ΄ôÜ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚246.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ƒ×1€‚‚(:õ •¹èÔŹ‘Þ% disk./.usageGB‚1172.0555953979492 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.51953125# disk.dm-0.out‚95776.09521484375! disk.nvme0n1.in‚ 7812.90234375 disk.sda.in‚0$ memory_percent‚68.64724159240723 disk.sda2.in‚0& disk.nvme0n1.out‚95776.09521484375# disk.nvme0c0n1.in‚ 7812.90234375 disk.sda.out‚ 108.17578125( proc.memory.availableMB‚ 5136.8359375! proc.memory.rssMB‚ 3344.328125* proc.memory.percent‚20.412158966064453 cpu‚25.2420818720955 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7812.90234375( disk.nvme0c0n1.out‚95776.09521484375 network.sent‚ 3.247328e+06 network.recv‚ 1.096013e+06) disk./.usagePercent‚78.65763442730336 disk.sda1.in‚0 disk.dm-1.out‚ 108.17578125 disk.sda3.in‚0 disk.sda3.out‚ 108.17578125 proc.cpu.threads‚178°Tlƒ‚(:ö ¤¹èÔŒÙÿÞ disk.sda.out‚ 108.67578125 disk.sda1.out‚0 network.sent‚ 3.253205e+06 network.recv‚ 1.100335e+06% disk./.usageGB‚1171.7694129943848 disk.dm-1.out‚ 108.67578125( disk.nvme0c0n1.out‚96073.59521484375 disk.sda1.in‚0$ memory_percent‚68.64571571350098 proc.cpu.threads‚17) disk./.usagePercent‚78.63842848607713 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚96073.59521484375 disk.sda3.in‚0 disk.sda3.out‚ 108.67578125( proc.memory.availableMB‚ 5137.0859375! proc.memory.rssMB‚ 3344.328125 disk.sda2.out‚0& disk.nvme0n1.out‚96073.59521484375* proc.memory.percent‚20.412158966064453 cpu‚25.24244220503777 disk.drbd1260.out‚ 0.51953125 disk.sda2.in‚0 disk.dm-0.in‚ 7812.90234375! disk.nvme0n1.in‚ 7812.90234375# disk.nvme0c0n1.in‚ 7812.90234375 disk.sda.in‚0Ñä©“„‚(:ˆ ¤¹èÔÓÕâä gpu.0.gpu‚65.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚255.496' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07]óy“…‚(:ˆ ³¹èÔ›ŠšÝ gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚239.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.9416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`…]3v†‚(:ë ³¹èÔ‚øØÞ disk.drbd1260.out‚ 0.5234375 disk.sda2.in‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚96448.458984375! proc.memory.rssMB‚ 3344.328125 network.sent‚ 3.258216e+06% disk./.usageGB‚1171.7666664123535 disk.dm-1.out‚ 109.53515625 disk.sda2.out‚0$ disk.nvme0n1.out‚96448.458984375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 7813.34765625! disk.dm-0.out‚96448.458984375! disk.nvme0n1.in‚ 7813.34765625 disk.sda3.out‚ 109.53515625# disk.nvme0c0n1.in‚ 7813.34765625 disk.sda.in‚0) disk./.usagePercent‚78.63824416065248 disk.sda.out‚ 109.53515625 disk.sda1.in‚0$ memory_percent‚96.78764343261719$ proc.memory.availableMB‚526.3125* proc.memory.percent‚20.412158966064453 cpu‚25.24254469858607 network.recv‚ 1.103828e+06/Ó•’x‡‚(:í ¹èÔøÊÑß disk.dm-1.out‚ 109.8828125 disk.sda2.in‚0 disk.dm-0.in‚ 7924.90234375! disk.dm-0.out‚96649.814453125! disk.nvme0n1.in‚ 7924.90234375 disk.sda.in‚0$ memory_percent‚96.78914546966553! proc.memory.rssMB‚ 3344.328125 network.recv‚ 1.107821e+06) disk./.usagePercent‚78.63824416065248 disk.drbd1260.in‚0$ disk.nvme0n1.out‚96649.814453125 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚20.412158966064453% disk./.usageGB‚1171.7666664123535 disk.drbd1260.out‚ 0.5234375 disk.sda2.out‚0 disk.sda3.out‚ 109.8828125& disk.nvme0c0n1.out‚96649.814453125( proc.memory.availableMB‚ 526.06640625 proc.cpu.threads‚17 network.sent‚ 3.263729e+06 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 7924.90234375 disk.sda.out‚ 109.8828125 cpu‚25.242831571091408u Ÿ.“ˆ‚(:ˆ ¹èÔòƒíß gpu.0.gpu‚42.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚330.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ylj‚(:’ ѹèÔ¾ÞûÝ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.178' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.43560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊSß3Š‚(:ö ѹèÔòõõÞ disk.sda2.out‚0# disk.dm-0.out‚97078.88037109375& disk.nvme0n1.out‚97078.88037109375% disk./.usageGB‚1171.7549362182617 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.52734375# memory_percent‚96.7952013015747( proc.memory.availableMB‚ 525.07421875* proc.memory.percent‚20.412158966064453 cpu‚25.243004783506354) disk./.usagePercent‚78.63745693748469 disk.dm-0.in‚ 7972.67578125( disk.nvme0c0n1.out‚97078.88037109375 proc.cpu.threads‚17 network.recv‚ 1.139485e+06 disk.sda.out‚ 110.41796875 disk.sda2.in‚0! disk.nvme0n1.in‚ 7972.67578125 disk.sda3.in‚0 disk.sda3.out‚ 110.41796875# disk.nvme0c0n1.in‚ 7972.67578125! proc.memory.rssMB‚ 3344.328125 network.sent‚ 3.283347e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 110.41796875 disk.drbd1260.in‚0KÄç‹‚(:ö à¹èÔžÃùà disk.sda.out‚ 111.01953125 disk.sda1.out‚0( proc.memory.availableMB‚ 525.79296875% disk./.usageGB‚1171.7666664123535 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.52734375 disk.sda2.out‚0# disk.dm-0.out‚97340.27490234375 disk.sda3.out‚ 111.01953125( disk.nvme0c0n1.out‚97340.27490234375 disk.sda.in‚0 network.recv‚ 1.144309e+06& disk.nvme0n1.out‚97340.27490234375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7972.67578125 disk.sda1.in‚0 disk.dm-1.out‚ 111.01953125$ memory_percent‚96.79081439971924* proc.memory.percent‚20.412158966064453 disk.dm-1.in‚0! proc.memory.rssMB‚ 3344.328125 cpu‚25.24327273794998 proc.cpu.threads‚17 network.sent‚ 3.289691e+06) disk./.usagePercent‚78.63824416065248 disk.sda2.in‚0 disk.dm-0.in‚ 7972.67578125! disk.nvme0n1.in‚ 7972.67578125^w’g“Œ‚(:ˆ à¹èÔ»¥áá gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚224.137' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.8274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®“2- _wandbruntime‚3663  _runtime‚3663ã -ÄŽ¾ trainer/global_step‚1249 % _timestamp‚1.7884848356961925e+09 _runtime‚3663.237963204 _step‚98 % lr-AdamW/pg1‚0.000978782664743832 % lr-AdamW/pg2‚0.000978782664743832öÎWÈ % _timestamp‚1.7884848356961925e+09 _runtime‚3663.237963204 _step‚98 % lr-AdamW/pg1‚0.000978782664743832 % lr-AdamW/pg2‚0.000978782664743832 trainer/global_step‚1249bQ»ïå2- _wandbruntime‚3663  _runtime‚3663"Æiòô'‘îO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884848356972818e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410225415267840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406411911232000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6996036 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57269276 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410225415267840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57295365 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚801 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17771392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10775356 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389440574126800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26279647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3804700772339 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3808267609675 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410221821009920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3809909777920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77497958470656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 _step‚99 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406411936659968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26279647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3813478607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30989635 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406411911232000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406411936659968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30989635 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4875448854528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚410221821009920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26279641 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26279641 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3813478607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚393245249471171 trainer/global_step‚1249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82373407325184 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809909777920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82373529220608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚393248841736475 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930jb%Cø'’òO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57269276 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410225415267840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57295365 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚801 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17771392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10775356 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389440574126800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26279647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3804700772339 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3808267609675 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410221821009920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3809909777920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77497958470656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 _step‚99 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406411936659968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26279647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3813478607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30989635 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406411911232000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406411936659968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30989635 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4875448854528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚410221821009920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26279641 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26279641 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3813478607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚393245249471171 trainer/global_step‚1249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82373407325184 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809909777920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82373529220608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚393248841736475 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884848356972818e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410225415267840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406411911232000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6996036 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448cÅô6Â2“- _wandbruntime‚3665  _runtime‚3665¦|ƒHŠ ”ùM WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389742245124000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31013927 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚796 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3665.456183995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57339933 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26299891 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31040036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3811258122497 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31013927 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406726527676416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406726502248448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57313818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17784307 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57313818 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406726527676416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚410543000884224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410543000884224 XèûìuFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3812904364544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17785106 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4879283829760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31040036 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 % _timestamp‚1.7884848379141748e+09 _step‚100 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7001520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3812904364544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 trainer/global_step‚1249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77559533444608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4879378711040 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3816473207808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3816473207808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚389742219696032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3807691284953 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚799 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57339933 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393553503246497 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26299891 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368§Ü«?'•ýM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3812904364544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 trainer/global_step‚1249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77559533444608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4879378711040 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3816473207808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3816473207808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚389742219696032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3807691284953 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚799 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57339933 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393553503246497 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26299891 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389742245124000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31013927 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚796 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3665.456183995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57339933 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26299891 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31040036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3811258122497 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31013927 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406726527676416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406726502248448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57313818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17784307 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57313818 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406726527676416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚410543000884224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410543000884224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3812904364544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17785106 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4879283829760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31040036 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 % _timestamp‚1.7884848379141748e+09 _step‚100 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7001520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448dYm2–- _wandbruntime‚3665  _runtime‚3665ÏÊ›¯Y—Ó 1loss/FBt2/assignment_loss‚0.19841652624469594 2loss/SRqqt1/assignment_loss‚2.2052723260389433 2loss/SRqqt2/assignment_loss‚1.8627697065886524 2loss/SRqqt4/assignment_loss‚1.0137276422673105 .loss/FRt3/detection_loss‚2.277738983077662 /loss/FBt1/detection_loss‚2.3433992006677262 epoch‚2 $ _timestamp‚1.788484837916527e+09 /loss/FRt1/assignment_loss‚3.921224656013342 1loss/FBt3/assignment_loss‚0.24309505242854357 1loss/FBt4/assignment_loss‚0.27041216493983355 .loss/FRt1/detection_loss‚3.155146743242557 /loss/FRt2/detection_loss‚2.2342385234725146 1loss/SRqqt3/detection_loss‚1.8504030843350021 .loss/FRt4/detection_loss‚1.798622042030096 0loss/SRqqt2/detection_loss‚2.998328633606434 _step‚101 1loss/SRqqt4/detection_loss‚1.1289961140802396 .loss/FBt3/detection_loss‚2.481647159602191 /loss/FBt4/detection_loss‚2.8872408909457072 0loss/FRt3/assignment_loss‚3.6304292572396144 1loss/FBt1/assignment_loss‚0.37755714884984054 0loss/SRqqt1/detection_loss‚2.818890257842011 /loss/FBt2/detection_loss‚0.9366759864530125 %loss/total_loss‚46.66841967291424 1loss/SRqqt3/assignment_loss‚1.613504319536415 /loss/FRt2/assignment_loss‚2.540192668084745 0loss/FRt4/assignment_loss‚1.8804905853271485 trainer/global_step‚1249 _runtime‚3665.457718137zj;Ü]˜× 1loss/FBt4/assignment_loss‚0.27041216493983355 .loss/FRt1/detection_loss‚3.155146743242557 /loss/FRt2/detection_loss‚2.2342385234725146 1loss/SRqqt3/detection_loss‚1.8504030843350021 .loss/FRt4/detection_loss‚1.798622042030096 0loss/SRqqt2/detection_loss‚2.998328633606434 _step‚101 1loss/SRqqt4/detection_loss‚1.1289961140802396 .loss/FBt3/detection_loss‚2.481647159602191 /loss/FBt4/detection_loss‚2.8872408909457072 0loss/FRt3/assignment_loss‚3.6304292572396144 1loss/FBt1/assignment_loss‚0.37755714884984054 0loss/SRqqt1/detection_loss‚2.818890257842011 /loss/FBt2/detection_loss‚0.9366759864530125 %loss/total_loss‚46.66841967291424 1loss/SRqqt3/assignment_loss‚1.613504319536415 /loss/FRt2/assignment_loss‚2.540192668084745 0loss/FRt4/assignment_loss‚1.8804905853271485 trainer/global_step‚1249 _runtime‚3665.457718137 1loss/FBt2/assignment_loss‚0.19841652624469594 2loss/SRqqt1/assignment_loss‚2.2052723260389433 2loss/SRqqt2/assignment_loss‚1.8627697065886524 2loss/SRqqt4/assignment_loss‚1.0137276422673105 .loss/FRt3/detection_loss‚2.277738983077662 /loss/FBt1/detection_loss‚2.3433992006677262 epoch‚2 $ _timestamp‚1.788484837916527e+09 /loss/FRt1/assignment_loss‚3.921224656013342 1loss/FBt3/assignment_loss‚0.24309505242854357eìáÙ“™‚(:ˆ ï¹èÔžý¼Ý gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚172.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°r qš‚(:æ ï¹èÔºôŠß disk.sda.in‚0! proc.memory.rssMB‚ 3344.328125 disk.sda2.out‚0! disk.nvme0n1.in‚ 7972.67578125 disk.sda3.out‚ 111.38671875* proc.memory.percent‚20.412158966064453 disk.sda2.in‚0 disk.dm-0.out‚ 97828.1015625) disk./.usagePercent‚78.63824236859973 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 526.51953125 cpu‚25.24352588636728 network.sent‚ 3.307825e+06 network.recv‚ 1.14812e+06 disk.dm-0.in‚ 7972.67578125" disk.nvme0n1.out‚ 97828.1015625# disk.nvme0c0n1.in‚ 7972.67578125 disk.sda.out‚ 111.38671875$ memory_percent‚96.78637981414795 proc.cpu.threads‚17% disk./.usageGB‚1171.7666397094727 disk.dm-1.in‚0 disk.dm-1.out‚ 111.38671875 disk.drbd1260.out‚0.53125 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 97828.1015625Ò“Xž›‚(:“ þ¹èÔý•¦á gpu.0.gpu‚58.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚212.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.532199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c>Ñjœ‚(:ß þ¹èԣǔâ disk.sda.in‚0 disk.dm-1.out‚ 111.76953125 disk.dm-0.out‚ 97981.34375! disk.nvme0n1.in‚ 7972.67578125* proc.memory.percent‚20.412158966064453 network.recv‚ 1.153794e+06# disk.nvme0c0n1.in‚ 7972.67578125" disk.nvme0c0n1.out‚ 97981.34375 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7972.67578125 disk.sda3.out‚ 111.76953125) disk./.usagePercent‚78.63824236859973% disk./.usageGB‚1171.7666397094727 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.out‚ 97981.34375$ memory_percent‚96.78945541381836& proc.memory.availableMB‚ 526.015625! proc.memory.rssMB‚ 3344.328125 network.sent‚ 3.38123e+06 disk.sda.out‚ 111.76953125 disk.sda1.out‚0 disk.drbd1260.out‚0.53125 disk.sda3.in‚0 cpu‚25.243909139359356 proc.cpu.threads‚17|%@‹u‚(:ê ºèÔ›¹ÿæ disk.sda2.out‚0# proc.memory.rssMB‚ 3344.33203125 cpu‚25.244194026444486 proc.cpu.threads‚17( disk./.usagePercent‚78.6382413445696 disk.sda.in‚0 disk.sda.out‚ 112.25390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8108.15625 disk.nvme0n1.in‚ 8108.15625% disk.nvme0n1.out‚98192.9482421875 network.recv‚ 1.165198e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.53515625 disk.sda3.out‚ 112.25390625# memory_percent‚96.7897891998291' proc.memory.availableMB‚ 525.9609375* proc.memory.percent‚20.412182807922363 network.sent‚ 3.390883e+06% disk./.usageGB‚1171.7666244506836 disk.nvme0c0n1.in‚ 8108.15625 disk.drbd1260.in‚0" disk.dm-0.out‚98192.9482421875 disk.sda3.in‚0' disk.nvme0c0n1.out‚98192.9482421875 disk.dm-1.out‚ 112.25390625!½¡“ž‚(:ˆ ºèÔ©ˆ…é gpu.0.gpu‚94.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚224.162' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.8324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä[¾“Ÿ‚(:ˆ œºèÔï”ûâ gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚238.321' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ·Ð¸v ‚(:ë œºèÔ…„âä$ disk./.usageGB‚1171.766716003418 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8108.15625 disk.nvme0n1.in‚ 8108.15625 network.sent‚ 3.401108e+06) disk./.usagePercent‚78.63824748875041 disk.drbd1260.out‚ 0.53515625 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 8108.15625$ memory_percent‚96.79601192474365( proc.memory.availableMB‚ 524.94140625" disk.dm-0.out‚98634.8798828125 disk.dm-1.out‚ 112.87109375% disk.nvme0n1.out‚98634.8798828125 disk.sda.out‚ 112.87109375" proc.memory.rssMB‚ 3344.3359375* proc.memory.percent‚20.412206649780273 proc.cpu.threads‚17 network.recv‚ 1.176887e+06 disk.sda3.out‚ 112.87109375' disk.nvme0c0n1.out‚98634.8798828125 disk.sda.in‚0 cpu‚25.244643202306158ëwó0r¡‚(:ç «ºèÔãÆ¡á disk.sda1.out‚0 network.sent‚ 3.406245e+06 disk.sda.in‚0 disk.sda.out‚ 113.3359375# memory_percent‚96.7885971069336% proc.memory.availableMB‚ 526.15625" proc.memory.rssMB‚ 3344.3359375 cpu‚25.244909792102735 disk.sda3.out‚ 113.3359375) disk./.usagePercent‚78.56603518755972 disk.drbd1260.out‚ 0.5390625 disk.dm-0.in‚ 8108.15625& disk.nvme0n1.out‚98982.58935546875 disk.dm-1.in‚0 disk.dm-1.out‚ 113.3359375* proc.memory.percent‚20.412206649780273% disk./.usageGB‚1170.6907005310059 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 network.recv‚ 1.180527e+06 disk.sda2.in‚0# disk.dm-0.out‚98982.58935546875 disk.nvme0n1.in‚ 8108.15625 disk.nvme0c0n1.in‚ 8108.15625( disk.nvme0c0n1.out‚98982.58935546875T$(±¢‚(:’ «ºèÔÚÓÚá gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.034' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.60679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿû™¡ž£‚(:“ ººèÔ€­Ý gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚152.792' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.558400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ònle¤‚(:Ú ººèÔÈŠ—Þ$ disk./.usageGB‚1170.690715789795 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 8108.15625% disk.nvme0c0n1.out‚99362.58203125 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 113.90625 disk.drbd1260.out‚ 0.5390625 network.sent‚ 3.411034e+06 network.recv‚ 1.183636e+06) disk./.usagePercent‚78.56603621158986 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 8108.15625" proc.memory.rssMB‚ 3344.3359375* proc.memory.percent‚20.412206649780273 disk.nvme0n1.in‚ 8108.15625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.out‚99362.58203125 cpu‚25.245232934387662 proc.cpu.threads‚17# disk.nvme0n1.out‚99362.58203125 disk.sda3.out‚ 113.90625 disk.sda.out‚ 113.90625 disk.sda2.in‚0$ memory_percent‚96.78864479064941' proc.memory.availableMB‚ 526.14843752ßËí`¥‚(:Õ ɺèÔá²ÊÞ network.recv‚ 1.188264e+06 disk.sda1.out‚0 disk.nvme0c0n1.out‚ 99845.375" proc.memory.rssMB‚ 3344.3359375 disk.sda.out‚ 114.21875 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚ 99845.375# disk.nvme0c0n1.in‚ 8235.69921875( proc.memory.availableMB‚ 526.14453125 cpu‚25.24537646080497 proc.cpu.threads‚17$ disk./.usageGB‚1170.814510345459 disk.dm-1.in‚0 disk.dm-1.out‚ 114.21875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8235.69921875! disk.nvme0n1.in‚ 8235.69921875 disk.nvme0n1.out‚ 99845.375) disk./.usagePercent‚78.57434416809103 disk.sda.in‚0 disk.drbd1260.out‚ 0.54296875 disk.sda3.in‚0 disk.sda3.out‚ 114.21875$ memory_percent‚96.78866863250732* proc.memory.percent‚20.412206649780273 network.sent‚ 3.417272e+06cÁ¢Ïž¦‚(:“ ɺèÔ´ ª„ gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚245.829' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.165800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÉí‡u§‚(:ê غèÔóô‹á( proc.memory.availableMB‚ 525.38671875" proc.memory.rssMB‚ 3344.3359375* proc.memory.percent‚20.412206649780273 cpu‚25.24568476635948 proc.cpu.threads‚17 disk.sda.out‚ 114.8515625! disk.nvme0n1.in‚ 8235.69921875# disk.nvme0n1.out‚99998.62890625 disk.sda3.in‚0% disk.nvme0c0n1.out‚99998.62890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 114.8515625% disk./.usageGB‚1170.9388046264648 disk.dm-0.out‚99998.62890625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.54296875 disk.sda.in‚0 network.sent‚ 3.429851e+06 network.recv‚ 1.205983e+06) disk./.usagePercent‚78.58268566157918 disk.dm-0.in‚ 8235.69921875 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚96.7932939529419 disk.sda3.out‚ 114.8515625# disk.nvme0c0n1.in‚ 8235.69921875 disk.dm-1.in‚0Ùå’““¨‚(:ˆ غèÔØÚûâ gpu.0.gpu‚97.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚45.0 gpu.0.powerWatts‚247.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ¿'ð2©- _wandbruntime‚3782  _runtime‚3782ù«vïǪÁ trainer/global_step‚1299 % _timestamp‚1.7884849549040108e+09 _runtime‚3782.445876832 _step‚102 & lr-AdamW/pg1‚0.0009732530580225712 & lr-AdamW/pg2‚0.0009732530580225712ýeÓ+Ë«Å & lr-AdamW/pg2‚0.0009732530580225712 trainer/global_step‚1299 % _timestamp‚1.7884849549040108e+09 _runtime‚3782.445876832 _step‚102 & lr-AdamW/pg1‚0.0009732530580225712f&Œ/2¬- _wandbruntime‚3782  _runtime‚3782F²†s¿­ïO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7270236 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85642674036736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96991744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3954253575047 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32207587 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3959666937344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3957820412383 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚422141487482368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 trainer/global_step‚1299 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32233670 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32233670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚829 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426101128991744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3963235767296 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27292147 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32207587 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59525817 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 % _timestamp‚1.7884849549059658e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18460396 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 _runtime‚3782.447898342 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚422141487482368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27292141 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27292141 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11190160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3X"¥}ù'®óO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7270233 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80575252269056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7270236 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85642674036736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96991744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3954253575047 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32207587 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3959666937344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3957820412383 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚422141487482368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 trainer/global_step‚1299 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32233670 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32233670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚829 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426101128991744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3963235767296 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27292147 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32207587 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59525817 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 % _timestamp‚1.7884849549059658e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18460396 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 _runtime‚3782.447898342 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚422141487482368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27292141 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27292141 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11190160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 B¿.ÿ&°ùM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27312397 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32257988 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3960810925205 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7275720 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85707911202816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚404825769556032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957244087661 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32257988 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚422456053070848 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884849572366245e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59570385 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426422308866048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11198406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5071113057792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 _runtime‚3784.779496268 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59544270 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27312391 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126076416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80636798145024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59570385 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3966230367232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18473291 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 trainer/global_step‚1299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59544270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27312397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚422456078498816 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32231879 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80636827243008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3966230367232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85708037279232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚422456053070848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3962661523968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5071210036224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚408783013643693 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11197574 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32231879 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232ÉX&Æ}±ýM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7275720 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85707911202816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚404825769556032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957244087661 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32257988 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚422456053070848 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884849572366245e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59570385 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426422308866048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11198406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5071113057792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 _runtime‚3784.779496268 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59544270 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27312391 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126076416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80636798145024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 PD2 „#†eviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59570385 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3966230367232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18473291 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 trainer/global_step‚1299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59544270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27312397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚422456078498816 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32231879 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80636827243008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3966230367232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85708037279232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚422456053070848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3962661523968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5071210036224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚408783013643693 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11197574 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32231879 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27312397 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32257988 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3960810925205 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136htA¦î2²- _wandbruntime‚3784  _runtime‚3784• þ Y³Ó /loss/FBt2/detection_loss‚0.7124923100281093 /loss/FBt4/detection_loss‚2.5764185140530267 epoch‚2 % _timestamp‚1.7884849572406764e+09 _step‚105 0loss/FBt3/assignment_loss‚0.3306700747376049 -loss/FRt1/detection_loss‚3.36275369540239 /loss/FRt3/detection_loss‚2.4814608615069162 1loss/SRqqt1/detection_loss‚2.5540099442005157 2loss/SRqqt2/assignment_loss‚2.2875623272077457 0loss/FRt4/assignment_loss‚1.8316252758460385 0loss/FBt1/assignment_loss‚0.3734175903652666 1loss/FBt4/assignment_loss‚0.23327435224723558 /loss/FRt2/detection_loss‚2.1602370904927906 /loss/FBt3/detection_loss‚2.7447519015382835 trainer/global_step‚1299 0loss/FRt2/assignment_loss‚2.7224996902725915 2loss/SRqqt1/assignment_loss‚1.8500587069852785 2loss/SRqqt3/assignment_loss‚1.6364338703355508 0loss/FBt2/assignment_loss‚0.2073612089627157 0loss/SRqqt3/detection_loss‚2.222116849924389 %loss/total_loss‚48.14847308796941 _runtime‚3784.782026728 /loss/FRt1/assignment_loss‚4.656548655950106 0loss/FRt3/assignment_loss‚3.8932618158204213 1loss/SRqqt4/assignment_loss‚1.042163590326497 /loss/FRt4/detection_loss‚1.7908803644490294 0loss/SRqqt2/detection_loss‚2.840704424155725 0loss/SRqqt4/detection_loss‚1.093533416589101 /loss/FBt1/detection_loss‚2.54423655657207266š¹]´× 1loss/SRqqt4/assignment_loss‚1.042163590326497 /loss/FRt4/detection_loss‚1.7908803644490294 0loss/SRqqt2/detection_loss‚2.840704424155725 0loss/SRqqt4/detection_loss‚1.093533416589101 /loss/FBt1/detection_loss‚2.5442365565720726 /loss/FBt2/detection_loss‚0.7124923100281093 /loss/FBt4/detection_loss‚2.5764185140530267 epoch‚2 % _timestamp‚1.7884849572406764e+09 _step‚105 0loss/FBt3/assignment_loss‚0.3306700747376049 -loss/FRt1/detection_loss‚3.36275369540239 /loss/FRt3/detection_loss‚2.4814608615069162 1loss/SRqqt1/detection_loss‚2.5540099442005157 2loss/SRqqt2/assignment_loss‚2.2875623272077457 0loss/FRt4/assignment_loss‚1.8316252758460385 0loss/FBt1/assignment_loss‚0.3734175903652666 1loss/FBt4/assignment_loss‚0.23327435224723558 /loss/FRt2/detection_loss‚2.1602370904927906 /loss/FBt3/detection_loss‚2.7447519015382835 trainer/global_step‚1299 0loss/FRt2/assignment_loss‚2.7224996902725915 2loss/SRqqt1/assignment_loss‚1.8500587069852785 2loss/SRqqt3/assignment_loss‚1.6364338703355508 0loss/FBt2/assignment_loss‚0.2073612089627157 0loss/SRqqt3/detection_loss‚2.222116849924389 %loss/total_loss‚48.14847308796941 _runtime‚3784.782026728 /loss/FRt1/assignment_loss‚4.656548655950106 0loss/FRt3/assignment_loss‚3.8932618158204213iõ/èc}µ‚(:ò çºèÔ´¹µâ# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.246135631222064 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8235.69921875$ memory_percent‚96.79067134857178( proc.memory.availableMB‚ 525.81640625 proc.cpu.threads‚17) disk./.usagePercent‚78.58266774105178 disk.sda.in‚0 disk.sda.out‚ 115.24609375 disk.dm-1.in‚0 disk.dm-1.out‚ 115.24609375 disk.drbd1260.out‚0.546875 disk.sda2.out‚0" disk.dm-0.out‚100361.033203125 network.recv‚ 1.20965e+06# disk.nvme0c0n1.in‚ 8235.69921875' disk.nvme0c0n1.out‚100361.033203125 disk.sda1.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 8235.69921875 disk.sda3.in‚0 disk.sda3.out‚ 115.24609375 network.sent‚ 3.43508e+06% disk./.usageGB‚1170.9385375976562% disk.nvme0n1.out‚100361.033203125é+¿^¶‚(:’ çºèÔîÕÇã gpu.0.gpu‚34.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚246.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µIõé’·‚(:‡ öºèÔù¹•Ý gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚51.0 gpu.0.powerWatts‚331.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.257 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ9†|¸‚(:ñ öºèÔùœýÝ' disk.nvme0c0n1.out‚100707.716796875 network.recv‚ 1.214663e+06 disk.dm-0.in‚ 8235.69921875! disk.nvme0n1.in‚ 8235.69921875 disk.sda3.in‚0( proc.memory.availableMB‚ 526.05078125 proc.cpu.threads‚17# disk./.usageGB‚1169.85347366333% disk.nvme0n1.out‚100707.716796875 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.546875 disk.sda2.in‚0* proc.memory.percent‚20.412230491638184 disk.sda.out‚ 115.52734375) disk./.usagePercent‚78.50984818998987" disk.dm-0.out‚100707.716796875 disk.sda3.out‚ 115.52734375# disk.nvme0c0n1.in‚ 8235.69921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 115.52734375 network.sent‚ 3.520628e+06 disk.sda2.out‚0$ memory_percent‚96.78924083709717# proc.memory.rssMB‚ 3344.33984375 cpu‚25.24640361698665y0¹‚(:ˆ …»èÔº¥ýÜ gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.904' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.9808 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°{ßðxº‚(:í …»èÔ ¤¨Ý# proc.memory.rssMB‚ 3344.33984375 cpu‚25.24655421143857 network.sent‚ 3.526936e+06 disk.sda.out‚ 116.1953125 disk.sda1.out‚0 disk.dm-1.out‚ 116.1953125 disk.drbd1260.in‚0# disk.dm-0.out‚101105.0576171875& disk.nvme0n1.out‚101105.0576171875$ memory_percent‚96.79079055786133 network.recv‚ 1.219712e+06) disk./.usagePercent‚78.50981670106314% disk./.usageGB‚1169.8530044555664 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 8401.671875 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 8401.671875( disk.nvme0c0n1.out‚101105.0576171875 disk.sda1.in‚0 disk.sda3.out‚ 116.1953125* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.55078125 disk.sda2.out‚0 disk.dm-0.in‚ 8401.671875& proc.memory.availableMB‚ 525.796875…ž»‚(:“ ”»èÔ’®žÞ gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚222.295' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.458999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ýZkw¼‚(:ì ”»èÔ©Ü¢à& disk.nvme0n1.out‚101376.7099609375 disk.sda3.in‚0( disk.nvme0c0n1.out‚101376.7099609375 disk.dm-1.out‚ 116.6953125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55078125 disk.dm-0.in‚ 8401.671875& proc.memory.availableMB‚ 526.046875 network.sent‚ 3.532833e+06! disk.nvme0c0n1.in‚ 8401.671875 disk.sda.out‚ 116.6953125# disk.dm-0.out‚101376.7099609375* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 network.recv‚ 1.223791e+06# disk./.usageGB‚1168.38822555542 disk.sda3.out‚ 116.6953125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚96.78926467895508 cpu‚25.246847098178687) disk./.usagePercent‚78.41151416004297 disk.nvme0n1.in‚ 8401.671875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3344.33984375à^.Ľ‚(:’ £»èÔ«ˆ¼ã gpu.0.gpu‚93.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚334.662' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.93239999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðsÏÅ{¾‚(:ð £»èÔøÈÈä( disk.nvme0c0n1.out‚101520.3935546875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 117.16015625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 8401.671875 network.sent‚ 3.538042e+06 disk.sda1.in‚0 disk.dm-0.in‚ 8401.671875& disk.nvme0n1.out‚101520.3935546875 disk.sda3.out‚ 117.16015625' proc.memory.availableMB‚ 526.0390625# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 disk.sda.in‚0 disk.sda.out‚ 117.16015625 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚101520.3935546875 disk.sda3.in‚0# memory_percent‚96.7893123626709) disk./.usagePercent‚78.41150929589982% disk./.usageGB‚1168.3881530761719 disk.drbd1260.out‚ 0.55078125 cpu‚25.24718423723781 proc.cpu.threads‚17 network.recv‚ 1.227278e+06! disk.nvme0c0n1.in‚ 8401.671875ç_™x¿‚(:í ²»èÔá…üà disk.sda3.out‚ 117.6015625! disk.nvme0c0n1.in‚ 8401.671875 disk.sda.out‚ 117.6015625 disk.dm-1.out‚ 117.6015625 disk.sda2.out‚0 network.recv‚ 1.230697e+06) disk./.usagePercent‚78.41832549649897 disk.dm-0.in‚ 8401.671875 disk.nvme0n1.in‚ 8401.671875 disk.sda1.in‚0 disk.sda2.in‚0# memory_percent‚96.7893123626709 cpu‚25.247622253069206& disk.nvme0n1.out‚101615.7060546875 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5546875' proc.memory.availableMB‚ 526.0390625 proc.cpu.threads‚17 network.sent‚ 3.542973e+06( disk.nvme0c0n1.out‚101615.7060546875 disk.sda1.out‚0# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184% disk./.usageGB‚1168.4897193908691# disk.dm-0.out‚101615.7060546875 disk.sda3.in‚0=¹›À‚(:’ ²»èÔ½ÿŸá gpu.0.gpu‚93.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚230.04' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.007999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÓf»“Á‚(:ˆ Á»èÔ꘵Þ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.093' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.2186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™(åv‚(:ë Á»èÔ©í·á disk.sda.out‚ 118.03515625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8541.15625 cpu‚25.247877257843346 proc.cpu.threads‚17 network.recv‚ 1.235481e+06( disk.nvme0c0n1.out‚102003.8701171875 disk.sda1.out‚0 disk.dm-1.out‚ 118.03515625 disk.drbd1260.out‚ 0.5546875# disk.dm-0.out‚102003.8701171875& disk.nvme0n1.out‚102003.8701171875 disk.sda3.out‚ 118.03515625) disk./.usagePercent‚78.41323657873333 disk.nvme0c0n1.in‚ 8541.15625 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚96.78936004638672% proc.memory.availableMB‚ 526.03125# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 network.sent‚ 3.549178e+06 disk.sda.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 8541.15625 disk.sda3.in‚0$ disk./.usageGB‚1168.413890838623ƒ'_=Â(:… лèÔŸö¸à gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚243.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.79 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥=s2zÄ‚(:ï лèÔª´¯á* proc.memory.percent‚20.412230491638184 cpu‚25.248200703691428 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375 disk.sda2.out‚0 disk.nvme0n1.in‚ 8541.15625& disk.nvme0n1.out‚102742.2255859375 disk.sda.out‚ 118.82421875 network.recv‚ 1.23961e+06) disk./.usagePercent‚78.43061897829318 disk.dm-0.in‚ 8541.15625 disk.sda3.in‚0$ memory_percent‚96.79090976715088 proc.cpu.threads‚17 network.sent‚ 3.555086e+06% disk./.usageGB‚1168.6729011535645 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚102742.2255859375 disk.sda3.out‚ 118.82421875 disk.nvme0c0n1.in‚ 8541.15625 disk.dm-1.out‚ 118.82421875 disk.sda2.in‚0( disk.nvme0c0n1.out‚102742.2255859375 disk.sda.in‚0( proc.memory.availableMB‚ 525.77734375# proc.memory.rssMB‚ 3344.33984375jsVÈ2Å- _wandbruntime‚3902  _runtime‚3902g³ûÅÆ¿ % lr-AdamW/pg1‚0.000967333989478863 % lr-AdamW/pg2‚0.000967333989478863 trainer/global_step‚1349 % _timestamp‚1.7884850752091503e+09 _runtime‚3902.750538023 _step‚106kXßÉÇà % lr-AdamW/pg1‚0.000967333989478863 % lr-AdamW/pg2‚0.000967333989478863 trainer/global_step‚1349 % _timestamp‚1.7884850752091503e+09 _runtime‚3902.750538023 _step‚106j臶2È- _wandbruntime‚3902  _runtime‚39026ÓO’õ'ÉïO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 _step‚107 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33425539 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419607673846800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4107373215091 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚437871012876800 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437871012876800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11604198 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28304647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11605010 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128186880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5258880177664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437871038304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4109424096768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4109424096768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441980436973568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7544436 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437871038304768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419607648418832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441980436973568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83652575165440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4112992926720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61756269 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5258979266560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4112992926720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11604198 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28304647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11605010 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128186880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5258880177664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437871038304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4109424096768 D>DeviceStatsMonitor.on_train_u±ãŒbatch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4109424096768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441980436973568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7544436 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437871038304768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419607648418832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441980436973568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83652575165440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4112992926720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61756269 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5258979266560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4112992926720 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61800837 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5262797727744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419909344844000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442298022576640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19162360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4106796890369 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28324891 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11612443 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19163195 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4110363727913 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 _runtime‚3905.064313983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28324897 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115987526656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7549920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88976790595584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83714121041408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88976918769152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442301616847872 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4112418683392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83714091943424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99075584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ÿzf9'ÍýM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128173568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33449831 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4112418683392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83714091943424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99075584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438185629321216 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438185603893248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61774722 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33475940 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11613275 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28324891 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚438185629321216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4112418683392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438185603893248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115987526656 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61800837 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5262797727744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419909344844000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442298022576640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19162360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4106796890369 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28324891 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11612443 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19163195 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4110363727913 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 _runtime‚3905.064313983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28324897 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115987526656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7549920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88976790595584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83714121041408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88976918769152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442301616847872 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4256926017799 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457859744955392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4262750086144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12018206 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5450325899776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438944557459295 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12019031 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚825 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438948149724599 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4259181256192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434691223706800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4262750086144 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434691198278832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34643491 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 trainer/global_step‚1399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92180194863616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.788485194812622e+09 _runtime‚4022.355297361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4259181256192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34669574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29317147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4253359180463 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453600563699200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19836839 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29317141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7818636 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86729868963840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457859744955392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101186048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5450224713728 _step‚111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453600589127168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457863339213312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453600563699200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457863339213312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453600589127168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86729839865856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29317141 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63986721 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34669574 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19837667 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63960632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92180064579584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7818633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0d¿¢eù'æóO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34669574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29317147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4253359180463 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453600563699200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19836839 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29317141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7818636 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86729868963840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457859744955392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101186048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5450224713728 _step‚111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453600589127168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457863339213312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453600563699200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457863339213312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453600589127168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86729839865856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29317141 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63986721 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34669574 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19837667 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63960632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92180064579584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7818633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29317147 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34643491 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4256926017799 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457859744955392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4262750086144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12018206 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5450325899776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438944557459295 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12019031 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚825 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438948149724599 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4259181256192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434691223706800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4262750086144 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434691198278832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34643491 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 trainer/global_step‚1399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92180194863616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.788485194812622e+09 _runtime‚4022.355297361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4259181256192oñÿ2ç- _wandbruntime‚4024  _runtime‚4024ËI©™ÕèùM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458177330558464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4259916530621 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128173568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34693892 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12026476 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4256349693077 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29337391 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34667783 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19850593 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚811 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92245437971456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚814 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64031289 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453915180143616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92245566145024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99075584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚434992869276032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86791385741824 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚434992894704000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64005174 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4262175842816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29337391 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64005174 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 % _timestamp‚1.7884851971591494e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64031289 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4262175842816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439249218969109 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4265744686080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7824117 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34667783 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453915154715648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _step‚112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7824120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34693892 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5454052229632 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453915180143616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29337397 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86791414839808©+ù³'éýM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34667783 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453915154715648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _step‚112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7824120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34693892 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5454052229632 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453915180143616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29337397 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86791414839808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458177330558464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4259916530621 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128173568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34693892 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12026476 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4256349693077 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29337391 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34667783 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19850593 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚811 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92245437971456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚814 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64031289 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453915180143616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92245566145024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99075584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚434992869276032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86791385741824 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚434992894704000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64005174 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4262175842816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29337391 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64005174 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 % _timestamp‚1.7884851971591494e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64031289 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4262175842816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439249218969109 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4265744686080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7824117 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0psã³h2ê- _wandbruntime‚4024  _runtime‚4024C»sƒ\ëÖ 2loss/SRqqt1/assignment_loss‚1.9001547824591398 2loss/SRqqt2/assignment_loss‚2.1835553031414747 0loss/FBt3/assignment_loss‚0.2873057824098737 1loss/FBt4/assignment_loss‚0.07661006199759868 .loss/FRt2/detection_loss‚2.252046996373919 &loss/total_loss‚45.565881349306025 /loss/FRt1/assignment_loss‚4.559031755273993 0loss/FRt2/assignment_loss‚2.5401871128927302 /loss/FRt3/assignment_loss‚3.649300780421809 1loss/SRqqt3/detection_loss‚2.2903011551458543 -loss/FRt1/detection_loss‚2.96438185301694 epoch‚2 2loss/SRqqt3/assignment_loss‚1.4858861943987542 2loss/SRqqt4/assignment_loss‚1.0465511492887336 1loss/FBt2/assignment_loss‚0.10403726794286176 1loss/SRqqt2/detection_loss‚2.8996698847040534 1loss/SRqqt4/detection_loss‚0.9483204404863282 trainer/global_step‚1399 % _timestamp‚1.7884851971629856e+09 /loss/FRt3/detection_loss‚2.2136922949238826 /loss/FRt4/detection_loss‚1.6809829426781526 /loss/FBt4/detection_loss‚2.3947579662005105 _runtime‚4024.704439022 _step‚113 /loss/FRt4/assignment_loss‚1.572868721352683 1loss/FBt1/assignment_loss‚0.34549200296576715 1loss/SRqqt1/detection_loss‚2.6656170074756327 /loss/FBt2/detection_loss‚0.8557644526267217 .loss/FBt3/detection_loss‚2.460387295112014 /loss/FBt1/detection_loss‚2.1889781460165976!;äŠ`ìÚ 2loss/SRqqt1/assignment_loss‚1.9001547824591398 2loss/SRqqt2/assignment_loss‚2.1835553031414747 0loss/FBt3/assignment_loss‚0.2873057824098737 1loss/FBt4/assignment_loss‚0.07661006199759868 .loss/FRt2/detection_loss‚2.252046996373919 &loss/total_loss‚45.565881349306025 /loss/FRt1/assignment_loss‚4.559031755273993 0loss/FRt2/assignment_loss‚2.5401871128927302 /loss/FRt3/assignment_loss‚3.649300780421809 1loss/SRqqt3/detection_loss‚2.2903011551458543 -loss/FRt1/detection_loss‚2.96438185301694 epoch‚2 2loss/SRqqt3/assignment_loss‚1.4858861943987542 2loss/SRqqt4/assignment_loss‚1.0465511492887336 1loss/FBt2/assignment_loss‚0.10403726794286176 1loss/SRqqt2/detection_loss‚2.8996698847040534 1loss/SRqqt4/detection_loss‚0.9483204404863282 trainer/global_step‚1399 % _timestamp‚1.7884851971629856e+09 /loss/FRt3/detection_loss‚2.2136922949238826 /loss/FRt4/detection_loss‚1.6809829426781526 /loss/FBt4/detection_loss‚2.3947579662005105 _runtime‚4024.704439022 _step‚113 /loss/FRt4/assignment_loss‚1.572868721352683 1loss/FBt1/assignment_loss‚0.34549200296576715 1loss/SRqqt1/detection_loss‚2.6656170074756327 /loss/FBt2/detection_loss‚0.8557644526267217 .loss/FBt3/detection_loss‚2.460387295112014 /loss/FBt1/detection_loss‚2.1889781460165976q+Üd,wí‚(:ì ×¼èÔ„áÛã disk.sda3.in‚0# proc.memory.rssMB‚ 3344.33984375( disk.nvme0c0n1.out‚105407.0244140625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 122.96875$ memory_percent‚96.78902626037598' proc.memory.availableMB‚ 526.0859375* proc.memory.percent‚20.412230491638184 network.sent‚ 3.686381e+06) disk./.usagePercent‚78.47738669065573 disk.sda.in‚0 disk.sda.out‚ 122.96875 disk.dm-1.out‚ 122.96875 disk.drbd1260.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚105407.0244140625 network.recv‚ 1.278788e+06 disk.sda1.out‚0 disk.dm-0.in‚ 8803.41796875 cpu‚25.2508461614214 proc.cpu.threads‚17$ disk./.usageGB‚1169.369773864746# disk.nvme0c0n1.in‚ 8803.41796875 disk.drbd1260.out‚ 0.57421875 disk.sda2.in‚0# disk.dm-0.out‚105407.0244140625! disk.nvme0n1.in‚ 8803.41796875Gm@Š‘î‚(:† ×¼èÔËú·å gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚134.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ru§î“ï‚(:ˆ æ¼èÔÆÕøÝ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-˜Xß}ð‚(:ò æ¼èÔÖ…ÝÞ disk.dm-1.out‚ 123.46875# proc.memory.rssMB‚ 3344.33984375 network.recv‚ 1.284415e+06( disk./.usagePercent‚78.4773879706934 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚105933.40283203125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 3.774944e+06% disk./.usageGB‚1169.3697929382324! disk.nvme0n1.in‚ 8803.41796875 disk.sda.out‚ 123.46875( proc.memory.availableMB‚ 525.83203125* proc.memory.percent‚20.412230491638184 cpu‚25.251148101149788 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.58203125 disk.dm-0.in‚ 8803.41796875$ disk.dm-0.out‚105933.40283203125 disk.sda3.out‚ 123.46875$ memory_percent‚96.79057598114014 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 8803.41796875) disk.nvme0c0n1.out‚105933.40283203125 disk.sda.in‚0 disk.dm-1.in‚0å¿}ñ‚(:ò õ¼èÔÿ¯—ß$ memory_percent‚96.79062366485596% disk./.usageGB‚1169.1830253601074 disk.sda3.in‚0 disk.sda3.out‚ 123.78125 disk.sda.out‚ 123.78125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.58203125( proc.memory.availableMB‚ 525.82421875# proc.memory.rssMB‚ 3344.33984375) disk./.usagePercent‚78.46485384181699! disk.nvme0n1.in‚ 8803.41796875) disk.nvme0c0n1.out‚106185.30517578125 disk.dm-1.out‚ 123.78125* proc.memory.percent‚20.412230491638184 cpu‚25.251379693309346 proc.cpu.threads‚17 disk.dm-0.in‚ 8803.41796875$ disk.dm-0.out‚106185.30517578125 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 network.sent‚ 3.77929e+06 network.recv‚ 1.289132e+06' disk.nvme0n1.out‚106185.30517578125# disk.nvme0c0n1.in‚ 8803.41796875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0k¤©\žò‚(:“ õ¼èÔß‹èà gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚236.975' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.394999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àŽA²žó‚(:“ „½èÔûÙœÝ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚242.569' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.513799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ä\ªsô‚(:è „½èÔ‡Ò€à! disk.nvme0c0n1.in‚ 8946.640625 disk.sda.in‚0 disk.sda2.in‚0" disk.dm-0.out‚106591.357421875 disk.nvme0n1.in‚ 8946.640625$ memory_percent‚96.79515361785889 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5859375% disk.nvme0n1.out‚106591.357421875 network.recv‚ 1.293433e+06% disk./.usageGB‚1169.2050895690918 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8946.640625 disk.sda3.in‚0 disk.sda3.out‚ 124.234375( proc.memory.availableMB‚ 525.08203125) disk./.usagePercent‚78.46633458939505' disk.nvme0c0n1.out‚106591.357421875 disk.sda.out‚ 124.234375 disk.sda1.out‚0 disk.dm-1.out‚ 124.234375# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.25160540629764 network.sent‚ 3.785147e+06¯(~Çõ‚(:’ “½èÔŠöñÜ gpu.0.gpu‚60.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚318.539' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚63.70779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rvS_xö‚(:í “½èÔ§ÙÞ disk.sda1.out‚0 disk.dm-1.out‚ 124.71484375% disk.nvme0n1.out‚106804.744140625 network.recv‚ 1.296885e+06! disk.nvme0c0n1.in‚ 8946.640625 disk.sda.in‚0 disk.sda.out‚ 124.71484375 disk.sda1.in‚0 disk.dm-0.in‚ 8946.640625" disk.dm-0.out‚106804.744140625 disk.nvme0n1.in‚ 8946.640625% disk./.usageGB‚1169.4362907409668 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 124.71484375$ memory_percent‚96.79217338562012' proc.memory.availableMB‚ 525.5703125# proc.memory.rssMB‚ 3344.33984375 disk.drbd1260.out‚ 0.5859375) disk./.usagePercent‚78.48185069403029' disk.nvme0c0n1.out‚106804.744140625 disk.drbd1260.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.412230491638184 cpu‚25.25204265379053 proc.cpu.threads‚17 network.sent‚ 3.792181e+06ŒƒN€÷‚(:õ ¢½èÔÀˬá disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.412230491638184' disk.nvme0n1.out‚107239.64892578125 disk.sda3.out‚ 125.08984375 disk.sda.out‚ 125.08984375 disk.drbd1260.out‚ 0.58984375 disk.dm-0.in‚ 8946.640625 proc.cpu.threads‚17 network.recv‚ 1.299614e+06) disk./.usagePercent‚78.47675230398589 disk.sda3.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚107239.64892578125( proc.memory.availableMB‚ 525.80859375# proc.memory.rssMB‚ 3344.33984375 cpu‚25.252293741809677% disk./.usageGB‚1169.3603210449219! disk.nvme0c0n1.in‚ 8946.640625) disk.nvme0c0n1.out‚107239.64892578125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 125.08984375 disk.sda2.in‚0# memory_percent‚96.7907190322876 network.sent‚ 3.794633e+06 disk.nvme0n1.in‚ 8946.640625sÇ«”“ø‚(:ˆ ¢½èÔ˜žïá gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚249.049' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8098 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…q²“ù‚(:ˆ ±½èÔ̽øÝ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚234.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.9178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬’A~ú‚(:ó ±½èÔÑþöà network.sent‚ 3.802682e+06$ disk./.usageGB‚1169.492618560791! disk.nvme0c0n1.in‚ 8946.640625 disk.sda.out‚ 125.63671875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.58984375 disk.dm-0.in‚ 8946.640625( proc.memory.availableMB‚ 525.80859375) disk./.usagePercent‚78.48563090128089 disk.dm-1.out‚ 125.63671875 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 125.63671875# memory_percent‚96.7907190322876# proc.memory.rssMB‚ 3344.33984375 cpu‚25.252469060464858 disk.sda2.in‚0 disk.nvme0n1.in‚ 8946.640625* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 network.recv‚ 1.30441e+06) disk.nvme0c0n1.out‚107500.00048828125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.dm-0.out‚107500.00048828125' disk.nvme0n1.out‚107500.00048828125ù$Kû‚(:ô À½èÔŠÙ¥â disk.sda1.out‚0 disk.dm-0.in‚ 9091.76171875 disk.sda3.in‚0$ memory_percent‚96.79229259490967# proc.memory.rssMB‚ 3344.33984375 cpu‚25.2528589340812% disk./.usageGB‚1169.4919242858887# disk.nvme0c0n1.in‚ 9091.76171875 disk.drbd1260.in‚0' disk.nvme0n1.out‚107982.72900390625( proc.memory.availableMB‚ 525.55078125* proc.memory.percent‚20.412230491638184 network.recv‚ 1.308772e+06) disk./.usagePercent‚78.48558430790963 disk.sda1.in‚0$ disk.dm-0.out‚107982.72900390625! disk.nvme0n1.in‚ 9091.76171875 proc.cpu.threads‚17) disk.nvme0c0n1.out‚107982.72900390625 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 126.1171875 disk.drbd1260.out‚0.59375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 126.1171875 network.sent‚ 3.806726e+06 disk.sda.out‚ 126.1171875€™cÆ“ü‚(:ˆ À½èÔ¨›Ìã gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚241.019' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.2038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢j«12ý- _wandbruntime‚4142  _runtime‚4142ò*ÑÇþÁ & lr-AdamW/pg1‚0.0009543475560293035 & lr-AdamW/pg2‚0.0009543475560293035 trainer/global_step‚1449 % _timestamp‚1.7884853150516129e+09 _runtime‚4142.592997576 _step‚1141D ËÿÅ & lr-AdamW/pg1‚0.0009543475560293035 & lr-AdamW/pg2‚0.0009543475560293035 trainer/global_step‚1449 % _timestamp‚1.7884853150516129e+09 _runtime‚4142.592997576 _step‚114r k‹2€- _wandbruntime‚4142  _runtime‚4142ʾy¬ª ïO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚469330114521600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66217173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35887526 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 _step‚115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449774748138832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469330114521600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35861443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12432065 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119798272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95449449435136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚454177660122003 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4406478820507 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20524134 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 L?DeviceStatsMonitor.on_tra®ÜËKin_batch_start/active.small_pool.freed‚35861443 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 trainer/global_step‚1449 _runtime‚4142.594547387 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30329641 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4412507245568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30329647 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469330139949568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30329641 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8092833 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35887526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8092836 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90700288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473742647195136 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4408938415616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4402911983171 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12431301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30329647 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20524901 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449774773566800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66191084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473739052937216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469330139949568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4408938415616%LØ™ù'‚óO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449774773566800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66191084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473739052937216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469330139949568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4408938415616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚469330114521600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66217173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35887526 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 _step‚115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449774748138832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469330114521600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35861443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12432065 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119798272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95449449435136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚454177660122003 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4406478820507 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20524134 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35861443 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 trainer/global_step‚1449 _runtime‚4142.594547387 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30329641 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4412507245568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30329647 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469330139949568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30329641 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8092833 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35887526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8092836 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90700288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473742647195136 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4408938415616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4402911983171 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12431301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30329647 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20524901s™$R(2ƒ- _wandbruntime‚4144  _runtime‚4144Leÿ&„ùM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚469644730966016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12439522 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30349897 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35885735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚469644730966016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4415501845504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66261741 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66235626 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450076419136032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4411933002240 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35885735 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5646175405056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4405902495785 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚469644705538048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4415501845504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 % _timestamp‚1.7884853172532547e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8098320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚766 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30349891 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66261741 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4411933002240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474056638540288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30349891 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89868679540224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90686976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚469644705538048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚769 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20537839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12440288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚454485913897329 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232:Ú,Fh…ýM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚454482321631817 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35911844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20538608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35911844 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35885735 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5646175405056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4405902495785 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚469644705538048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4415501845504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 % _timestamp‚1.7884853172532547e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8098320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚766 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30349891 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66261741 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4411933002240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474056638540288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30349891 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89868679540224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90686976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4Ôžík›69644705538048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚769 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20537839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12440288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚454485913897329 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚469644730966016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12439522 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30349897 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35885735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚469644730966016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4415501845504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66261741 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66235626 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450076419136032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4411933002240 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489618360919040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37105478 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489621955176960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚770 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚485059690771968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92884427462656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489618360919040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464858297998832 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚773 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12847579 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31342147 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68447625 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5835431142912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21213842 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 % _timestamp‚1.7884854352759469e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚464858323426800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚485059665344000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4558695575040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68421536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21214615 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8367033 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0œI7²ažóO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37079395 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚485059665344000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚485059690771968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92884427462656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489618360919040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464858297998832 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚773 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12847579 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31342147 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68447625 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5835431142912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21213842 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 % _timestamp‚1.7884854352759469e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚464858323426800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚485059665344000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4558695575040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68421536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21214615 SGDØ8˜eviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8367033 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4262.818595957 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37105478 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37079395 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98719765808128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489621955176960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31342141 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4562264404992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4552464785879 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4558695575040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98719887703552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚469414355050015 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 trainer/global_step‚1499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489618360919040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37105478 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489621955176960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚770 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4561690161664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4565259004928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92946002436608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37129796 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚485374256360448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31362391 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8372520 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68492193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839284733440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68466078 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31362397 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚485374281788416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21227532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4555455298493 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92784128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469715424294525 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 _runtime‚ 4265.08076421 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚775 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8372520 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68492193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839284733440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68466078 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31362397 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚485374281788416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21227532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4555455298493 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92784128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469715424294525 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 _runtime‚ 4265.08076421 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚775 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4561690161664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4565259004928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92946002436608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37129796 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚485374256360448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232xΨ²2¢- _wandbruntime‚4265  _runtime‚4265)ðˆ\[£Õ epoch‚2 0loss/FRt4/assignment_loss‚1.9258794960521517 1loss/SRqqt2/assignment_loss‚2.216945949821703 1loss/FBt3/assignment_loss‚0.10752629834080706 /loss/FRt4/detection_loss‚1.7617855243739629 0loss/SRqqt2/detection_loss‚2.825981418932638 /loss/FBt1/detection_loss‚2.4051702383792763 _runtime‚ 4265.0826708 2loss/SRqqt3/assignment_loss‚1.6684940078888426 /loss/FRt1/detection_loss‚3.0626562925485463 _step‚121 1loss/FBt4/assignment_loss‚0.08552609856997151 0loss/FRt1/assignment_loss‚4.5702230380131645 /loss/FBt2/detection_loss‚0.6991793904523018 trainer/global_step‚1499 % _timestamp‚1.7884854375415075e+09 %loss/total_loss‚46.82920688038865 0loss/FRt2/assignment_loss‚2.5256145390463463 1loss/FBt2/assignment_loss‚0.21602702163884716 /loss/FRt3/detection_loss‚2.3759817730635406 1loss/SRqqt1/detection_loss‚2.8694060802459718 /loss/FBt4/detection_loss‚2.7226490068894167 .loss/FRt2/detection_loss‚2.274944999474058 /loss/FRt3/assignment_loss‚3.972627554833889 2loss/SRqqt4/assignment_loss‚1.1586218479177142 0loss/FBt1/assignment_loss‚0.1294227208633019 1loss/SRqqt4/detection_loss‚1.0606283812445376 1loss/SRqqt1/assignment_loss‚1.883910301164724 1loss/SRqqt3/detection_loss‚1.8612893628464504 /loss/FBt3/detection_loss‚2.4487155377864838jæ*_¤Ù %loss/total_loss‚46.82920688038865 0loss/FRt2/assignment_loss‚2.5256145390463463 1loss/FBt2/assignment_loss‚0.21602702163884716 /loss/FRt3/detection_loss‚2.3759817730635406 1loss/SRqqt1/detection_loss‚2.8694060802459718 /loss/FBt4/detection_loss‚2.7226490068894167 .loss/FRt2/detection_loss‚2.274944999474058 /loss/FRt3/assignment_loss‚3.972627554833889 2loss/SRqqt4/assignment_loss‚1.1586218479177142 0loss/FBt1/assignment_loss‚0.1294227208633019 1loss/SRqqt4/detection_loss‚1.0606283812445376 1loss/SRqqt1/assignment_loss‚1.883910301164724 1loss/SRqqt3/detection_loss‚1.8612893628464504 /loss/FBt3/detection_loss‚2.4487155377864838 epoch‚2 0loss/FRt4/assignment_loss‚1.9258794960521517 1loss/SRqqt2/assignment_loss‚2.216945949821703 1loss/FBt3/assignment_loss‚0.10752629834080706 /loss/FRt4/detection_loss‚1.7617855243739629 0loss/SRqqt2/detection_loss‚2.825981418932638 /loss/FBt1/detection_loss‚2.4051702383792763 _runtime‚ 4265.0826708 2loss/SRqqt3/assignment_loss‚1.6684940078888426 /loss/FRt1/detection_loss‚3.0626562925485463 _step‚121 1loss/FBt4/assignment_loss‚0.08552609856997151 0loss/FRt1/assignment_loss‚4.5702230380131645 /loss/FBt2/detection_loss‚0.6991793904523018 trainer/global_step‚1499 % _timestamp‚1.7884854375415075e+09yv¢“¥‚(:ˆ ǾèÔ®”ºß gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚230.293' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.0586 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ˆò(v¦‚(:ë ǾèÔ¥¢ìß network.sent‚ 3.938072e+06 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚110674.80859375 disk.sda3.out‚ 130.1171875# proc.memory.rssMB‚ 3344.33984375 cpu‚25.25509775945241 proc.cpu.threads‚17 network.recv‚ 1.34675e+06 disk.sda.in‚0 disk.sda.out‚ 130.1171875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.61328125 disk.sda2.out‚0! disk.nvme0n1.in‚ 9354.39453125) disk./.usagePercent‚78.44859915544549# disk.nvme0c0n1.in‚ 9354.39453125& disk.nvme0c0n1.out‚110674.80859375 disk.sda1.in‚0 disk.dm-1.out‚ 130.1171875 disk.sda3.in‚0# memory_percent‚96.7923641204834' proc.memory.availableMB‚ 525.5390625$ disk./.usageGB‚1168.940818786621 disk.dm-0.in‚ 9354.39453125! disk.dm-0.out‚110674.80859375* proc.memory.percent‚20.412230491638184: “§‚(:ˆ Ö¾èÔ±ÐÔà gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚239.679' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.9358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýE¨Y{¨‚(:ð Ö¾èÔëæá# disk.nvme0c0n1.in‚ 9354.39453125 disk.dm-1.out‚ 130.41796875 disk.drbd1260.in‚0% disk.nvme0n1.out‚111260.779296875 network.sent‚ 4.024778e+06 network.recv‚ 1.352722e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 130.41796875 proc.cpu.threads‚17# disk./.usageGB‚1169.01957321167' disk.nvme0c0n1.out‚111260.779296875 disk.sda.out‚ 130.41796875 disk.dm-1.in‚0! disk.nvme0n1.in‚ 9354.39453125% proc.memory.availableMB‚ 525.28125# proc.memory.rssMB‚ 3344.33984375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.6171875 disk.sda2.out‚0 disk.dm-0.in‚ 9354.39453125" disk.dm-0.out‚111260.779296875$ memory_percent‚96.79393768310547* proc.memory.percent‚20.412230491638184 cpu‚25.255400729454042) disk./.usagePercent‚78.45388443098987y0¡#“©‚(:ˆ å¾èÔ±…àÝ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.427' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{goEª‚(:ì å¾èÔã›…ß) disk./.usagePercent‚78.45263025007962# disk.nvme0c0n1.in‚ 9354.39453125 disk.sda1.out‚0 disk.dm-1.out‚ 130.796875 disk.dm-0.in‚ 9354.39453125$ memory_percent‚96.79241180419922* proc.memory.percent‚20.412230491638184 network.recv‚ 1.357681e+06' disk.nvme0c0n1.out‚ÿ}£Ú2111633.482421875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.dm-0.out‚111633.482421875 disk.sda3.in‚0 disk.sda3.out‚ 130.796875% disk./.usageGB‚1169.0008850097656 disk.sda.out‚ 130.796875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.6171875 disk.sda2.in‚0! disk.nvme0n1.in‚ 9354.39453125# proc.memory.rssMB‚ 3344.33984375 cpu‚25.255700228411524 network.sent‚ 4.030901e+06 disk.sda.in‚0% disk.nvme0n1.out‚111633.482421875% proc.memory.availableMB‚ 525.53125 proc.cpu.threads‚17ø`€w«‚(:ì ô¾èÔ’§¥â# disk.nvme0c0n1.in‚ 9354.39453125' disk.nvme0c0n1.out‚111914.509765625 disk.drbd1260.out‚ 0.62109375% disk.nvme0n1.out‚111914.509765625" proc.memory.availableMB‚525.25 cpu‚25.25592677800129 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 131.1953125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 131.1953125 disk.dm-0.in‚ 9354.39453125" disk.dm-0.out‚111914.509765625$ memory_percent‚96.79412841796875 network.sent‚ 4.036963e+06 network.recv‚ 1.362209e+06 disk.sda1.in‚0 disk.sda2.in‚0% disk./.usageGB‚1169.5243225097656 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 9354.39453125 disk.sda3.in‚0 disk.sda3.out‚ 131.1953125# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184) disk./.usagePercent‚78.48775857989794ä•ž¬‚(:“ ô¾èÔƒ´€ã gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚251.099' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.219800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Tc­‚(:’ ƒ¿èÔ—¼ÀÝ gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚338.636' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.72720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍaZ¢€®‚(:õ ƒ¿èÔó‚€à( proc.memory.availableMB‚ 525.24609375# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 network.sent‚ 4.041993e+06 disk.drbd1260.in‚0 disk.dm-0.in‚ 9492.87109375! disk.nvme0n1.in‚ 9492.87109375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 131.56640625) disk./.usagePercent‚78.49194763118071$ disk./.usageGB‚1169.586742401123 disk.sda2.in‚0 disk.dm-1.in‚0 cpu‚25.256217017624934 disk.sda3.out‚ 131.56640625 network.recv‚ 1.366018e+06 disk.drbd1260.out‚ 0.62109375% disk.nvme0n1.out‚112069.865234375' disk.nvme0c0n1.out‚112069.865234375 disk.sda1.out‚0 disk.dm-1.out‚ 131.56640625$ memory_percent‚96.79415225982666 disk.sda2.out‚0" disk.dm-0.out‚112069.865234375# disk.nvme0c0n1.in‚ 9492.87109375 disk.sda1.in‚0ÊѶt¯‚(:é ’¿èÔ‚¼ôÞ disk.dm-0.in‚ 9492.87109375# disk.dm-0.out‚112393.2568359375& disk.nvme0n1.out‚112393.2568359375 disk.sda3.out‚ 131.96875' proc.memory.availableMB‚ 525.7421875* proc.memory.percent‚20.412230491638184( disk.nvme0c0n1.out‚112393.2568359375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚96.79112434387207 cpu‚25.256378322248548 network.recv‚ 1.369454e+06) disk./.usagePercent‚78.50456547452082 disk.drbd1260.in‚0 disk.drbd1260.out‚0.625! disk.nvme0n1.in‚ 9492.87109375 proc.cpu.threads‚17 disk.dm-1.out‚ 131.96875 network.sent‚ 4.047007e+06$ disk./.usageGB‚1169.774757385254 disk.sda.in‚0 disk.sda.out‚ 131.96875 disk.sda1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 3344.33984375# disk.nvme0c0n1.in‚ 9492.87109375 disk.sda2.in‚0%ßñ°‚(:’ ’¿èÔÛ›­á gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚224.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.989999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÞ…~±‚(:ó ¡¿èÔŽ±¥à disk.sda2.out‚0 disk.dm-0.in‚ 9492.87109375# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.256622052833468) disk./.usagePercent‚78.50111218889145# disk.nvme0c0n1.in‚ 9492.87109375( disk.nvme0c0n1.out‚112796.7333984375 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 4.053279e+06$ disk./.usageGB‚1169.723300933838! disk.nvme0n1.in‚ 9492.87109375 disk.sda3.out‚ 132.35546875 disk.sda2.in‚0( proc.memory.availableMB‚ 525.48828125& disk.nvme0n1.out‚112796.7333984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 132.35546875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.625# disk.dm-0.out‚112796.7333984375$ memory_percent‚96.79267406463623 network.recv‚ 1.374108e+06 disk.sda.out‚ 132.35546875 disk.sda1.out‚0eû¸²‚(:’ ¡¿èÔ°¼â gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.034' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.60679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”e“󑳂(:† °¿èÔŸ˜æÝ gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚49.0 gpu.0.powerWatts‚323.707' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.7414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ǻ©‚´‚(:÷ °¿èÔ—ì†à$ memory_percent‚96.79570198059082* proc.memory.percent‚20.412230491638184 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.62890625$ disk.dm-0.out‚113215.86962890625# proc.memory.rssMB‚ 3344.33984375 cpu‚25.256826903112124 proc.cpu.threads‚17 network.sent‚ 4.059178e+06 disk.sda.out‚ 132.7734375 disk.drbd1260.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 524.9921875 network.recv‚ 1.378562e+06 disk.sda3.out‚ 132.7734375# disk.nvme0c0n1.in‚ 9492.87109375) disk.nvme0c0n1.out‚113215.86962890625 disk.dm-1.in‚0 disk.dm-1.out‚ 132.7734375 disk.sda2.out‚0 disk.dm-0.in‚ 9492.87109375) disk./.usagePercent‚78.51788989865543$ disk./.usageGB‚1169.973300933838 disk.sda.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 9492.87109375' disk.nvme0n1.out‚113215.86962890625xÏû2µ- _wandbruntime‚4382  _runtime‚4382©KtǶÁ trainer/global_step‚1549 % _timestamp‚1.7884855546911077e+09 _runtime‚4382.232660343 _step‚122 & lr-AdamW/pg1‚0.0009398667264508672 & lr-AdamW/pg2‚0.0009398667264508672†¢/Ë·Å _runtime‚4382.232660343 _step‚122 & lr-AdamW/pg1‚0.0009398667264508672 & lr-AdamW/pg2‚0.0009398667264508672 trainer/global_step‚1549 % _timestamp‚1.7884855546911077e+09zú;·j2¸- _wandbruntime‚4382  _runtime‚4382ä•Þà÷'¹ñO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505497668900864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95961750359040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70678077 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 % _timestamp‚1.7884855546929433e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32354647 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚773 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38323430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13261650 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505501263158784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101990207533056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6028364376576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8641236 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505497668900864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6028457174016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 trainer/global_step‚1549 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚479941873286800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4712021564416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500789241594368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484643865447419 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4705584425923 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484647457712723 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13262420 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32354647 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500789216166400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38323430 _step‚123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500789216166400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21903656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8641233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4702017588587 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505501263158784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4712021564416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101990085637632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500789241594368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21902883 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319¹ Ðõû'ºõO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13262420 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32354647 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500789216166400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38323430 _step‚123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500789216166400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21903656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8641233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4702017588587 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505501263158784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4712021564416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101990085637632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500789241594368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21902883 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505497668900864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95961750359040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70678077 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 % _timestamp‚1.7884855546929433e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32354647 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚773 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38323430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13261650 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505501263158784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101990207533056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6028364376576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8641236 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505497668900864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6028457174016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 trainer/global_step‚1549 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚479941873286800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4712021564416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500789241594368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484643865447419 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4705584425923 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484647457712723 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136{ntnM2»- _wandbruntime‚4384  _runtime‚4384—z.¹¾¼úM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505818848775168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38321639 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70696530 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32374897 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4715016164352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96023296235008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96023267137024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505818848775168 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6032315922432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4708574938745 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 _runtime‚4384.622021516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501103807182848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38321639 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32374897 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4705008101201 trainer/global_step‚1549 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21916645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚773 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484948526957233 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102055488178176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501103807182848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70722645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13270701 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 _step‚124 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚480243544284000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8646717 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501103832610816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4711447321088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505818848775168 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6032315922432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4708574938745 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 _runtime‚4384.622021516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501103807182848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38321639 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32374897 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4705008101201 trainer/global_step‚1549 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21916645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚773 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484948526957233 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102055488178176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501103807182848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70722645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13270701 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 _step‚124 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94881280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚480243544284000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8646717 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501103832610816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4711447321088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505818848775168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38321639 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70696530 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32374897 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4715016164352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96023296235008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96023267137024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521376976882688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4499.831992704 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521380571140608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39541382 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4858209893888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚516518792416768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22591931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99039044157440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39515299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4861778723840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33367147 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33367141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99039015059456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8915436 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33367141 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105260373283840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚521376976882688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4851570391295 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33367147 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4858209893888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13675717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6221329126400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚499876968110127 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚778 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚499880560375431 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72908529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72882440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4855137228631 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4861778723840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521376976882688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4499.831992704 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521380571140608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39541382 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4858209893888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚516518792416768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22591931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99039044157440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39515299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4861778723840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33367147 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33367141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99039015059456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8915436 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33367141 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105260373283840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚521376976882688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4851570391295 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33367147 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4858209893888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13675717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6221329126400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚499876968110127 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚778 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚499880560375431 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72908529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72882440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4855137228631 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4861778723840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39539591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33387397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6225183307776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4861204480512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516833358005248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8920917 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99100590033408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92784128 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39565700 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105325651459072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99100560935424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33387397 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6225090523648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22604793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13684646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚770 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚521698156756992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516833383433216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72953097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚773 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882112 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4861204480512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72926982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4864773323776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884856744797957e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4864773323776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚495327094144000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8920920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4502.021963723 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448KÉ:ÛYÙ€N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22604793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13684646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚770 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚521698156756992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516833383433216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72953097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚773 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882112 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4861204480512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72926982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4864773323776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884856744797957e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4864773323776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚495327094144000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8920920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4502.021963723 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ƒúu«­ 52238108672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500181629619941 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚495327068716032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72926982 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39539591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33387397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6225183307776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4861204480512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516833358005248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8920917 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99100590033408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92784128 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39565700 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105325651459072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99100560935424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33387397 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6225090523648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120€]CÑ2Ú- _wandbruntime‚4502  _runtime‚4502OÎd[ÛÕ 0loss/FRt4/assignment_loss‚1.6951280107864968 2loss/SRqqt3/assignment_loss‚1.9284289125353098 1loss/SRqqt4/assignment_loss‚1.056329595161891 1loss/FBt3/assignment_loss‚0.34145723362735225 .loss/FBt1/detection_loss‚1.886890249543412 0loss/SRqqt1/detection_loss‚2.466462177165011 1loss/SRqqt2/detection_loss‚3.4326731814787936 1loss/SRqqt3/detection_loss‚2.1691978597505526 trainer/global_step‚1599 1loss/FBt2/assignment_loss‚0.08248912910728656 /loss/FBt3/detection_loss‚2.4468820560723543 /loss/FRt3/detection_loss‚2.4216922630282007 /loss/FRt4/detection_loss‚1.6876159034382838 epoch‚2 _step‚129 0loss/FRt2/assignment_loss‚2.7482638617595994 1loss/SRqqt4/detection_loss‚1.1504250055275345 &loss/total_loss‚47.553338564291856 2loss/SRqqt2/assignment_loss‚2.0862815242547255 .loss/FRt1/detection_loss‚3.295076347887516 .loss/FBt2/detection_loss‚0.780778051236659 % _timestamp‚1.7884856744823325e+09 _runtime‚4502.023512795 0loss/FRt3/assignment_loss‚4.3833310968735635 0loss/FBt1/assignment_loss‚0.3840555599611086 .loss/FBt4/detection_loss‚2.825550675392151 /loss/FRt1/assignment_loss‚4.163759605586529 2loss/SRqqt1/assignment_loss‚1.7014815764541322 1loss/FBt4/assignment_loss‚0.17843761336019573 .loss/FRt2/detection_loss‚2.240651074303202Á&|á`ÜÚ .loss/FBt2/detection_loss‚0.780778051236659 % _timestamp‚1.7884856744823325e+09 _runtime‚4502.023512795 0loss/FRt3/assignment_loss‚4.3833310968735635 0loss/FBt1/assignment_loss‚0.3840555599611086 .loss/FBt4/detection_loss‚2.825550675392151 /loss/FRt1/assignment_loss‚4.163759605586529 2loss/SRqqt1/assignment_loss‚1.7014815764541322 1loss/FBt4/assignment_loss‚0.17843761336019573 .loss/FRt2/detection_loss‚2.240651074303202 0loss/FRt4/assignment_loss‚1.6951280107864968 2loss/SRqqt3/assignment_loss‚1.9284289125353098 1loss/SRqqt4/assignment_loss‚1.056329595161891 1loss/FBt3/assignment_loss‚0.34145723362735225 .loss/FBt1/detection_loss‚1.886890249543412 0loss/SRqqt1/detection_loss‚2.466462177165011 1loss/SRqqt2/detection_loss‚3.4326731814787936 1loss/SRqqt3/detection_loss‚2.1691978597505526 trainer/global_step‚1599 1loss/FBt2/assignment_loss‚0.08248912910728656 /loss/FBt3/detection_loss‚2.4468820560723543 /loss/FRt3/detection_loss‚2.4216922630282007 /loss/FRt4/detection_loss‚1.6876159034382838 epoch‚2 _step‚129 0loss/FRt2/assignment_loss‚2.7482638617595994 1loss/SRqqt4/detection_loss‚1.1504250055275345 &loss/total_loss‚47.553338564291856 2loss/SRqqt2/assignment_loss‚2.0862815242547255 .loss/FRt1/detection_loss‚3.295076347887516ò@“Ý‚(:ˆ ·ÀèÔÅöÝ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.496' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]o«µ‡Þ‚(:ü ·ÀèÔⲡÞ network.recv‚ 1.418593e+06% disk./.usageGB‚1169.3653450012207) disk.nvme0c0n1.out‚116377.79248046875 disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚10004.22265625$ memory_percent‚96.79591655731201( proc.memory.availableMB‚ 524.95703125$ disk.nvme0c0n1.in‚10004.22265625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.64453125" disk.nvme0n1.in‚10004.22265625 disk.sda3.out‚ 136.6953125# proc.memory.rssMB‚ 3344.33984375 cpu‚25.258389519491303 network.sent‚ 4.260006e+06) disk./.usagePercent‚78.47708946590848 disk.sda.in‚0 disk.sda.out‚ 136.6953125 disk.dm-1.out‚ 136.6953125$ disk.dm-0.out‚116377.79248046875' disk.nvme0n1.out‚116377.79248046875 disk.sda3.in‚0êùcß‚(:’ ÆÀèÔ¨øÆã gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.092' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.81840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.«aK…à‚(:ú ÆÀèÔ« Îå% disk./.usageGB‚1169.3653450012207 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚10004.22265625$ disk.dm-0.out‚116723.55419921875' disk.nvme0n1.out‚116723.55419921875) disk.nvme0c0n1.out‚116723.55419921875 network.sent‚ 4.337789e+06" disk.nvme0n1.in‚10004.22265625* proc.memory.percent‚20.412230491638184 cpu‚25.258712373520645 proc.cpu.threads‚17) disk./.usagePercent‚78.47708946590848 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.6484375 disk.sda3.in‚0$ disk.nvme0c0n1.in‚10004.22265625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.423752e+06 disk.dm-1.out‚ 137.2109375 disk.sda3.out‚ 137.2109375 disk.sda.out‚ 137.2109375# memory_percent‚96.7975378036499( proc.memory.availableMB‚ 524.69140625# proc.memory.rssMB‚ 3344.33984375ë-Ðœá‚(:‘ ÕÀèÔçªÝ gpu.0.gpu‚16.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚326.583' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.31660000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09økŒâ‚(:ö ÕÀèÔ‡†³Þ) disk.nvme0c0n1.out‚117127.05810546875 disk.sda.out‚ 137.953125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚10004.22265625' proc.memory.availableMB‚ 524.6796875 network.recv‚ 1.42853e+06 disk.sda3.out‚ 137.953125 disk.dm-1.out‚ 137.953125 disk.drbd1260.out‚ 0.6484375 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚117127.05810546875$ memory_percent‚96.79760932922363 disk.dm-0.in‚10004.22265625' disk.nvme0n1.out‚117127.05810546875# proc.memory.rssMB‚ 3344.33984375 cpu‚25.258928472702497 network.sent‚ 4.342069e+06) disk./.usagePercent‚78.47709125796122 disk.sda3.in‚0$ disk.nvme0c0n1.in‚10004.22265625 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17% disk./.usageGB‚1169.36537170410162ÞIê…ã‚(:ú äÀèÔ¡ˆ—á) disk.nvme0c0n1.out‚117448.74462890625 disk.sda.in‚0$ memory_percent‚96.79450988769531 proc.cpu.threads‚17% disk./.usageGB‚1169.3653945922852 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 138.24609375 disk.sda.out‚ 138.24609375 cpu‚25.25913808594383) disk./.usagePercent‚78.47709279400642$ disk.dm-0.out‚117448.74462890625" disk.nvme0n1.in‚10004.22265625$ proc.memory.availableMB‚525.1875# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 network.recv‚ 1.433091e+06 disk.sda1.in‚0 disk.dm-1.out‚ 138.24609375 disk.drbd1260.out‚ 0.65234375 disk.sda2.in‚0 disk.dm-0.in‚10004.22265625' disk.nvme0n1.out‚117448.74462890625$ disk.nvme0c0n1.in‚10004.22265625 network.sent‚ 4.347988e+06 disk.dm-1.in‚0 disk.sda2.out‚0’)·žä‚(:“ äÀèÔ¦”»á gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.040600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;M¼ “å‚(:ˆ óÀèÔÚÏøÞ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚143.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÓ-zˆæ‚(:ý óÀèԆşß$ memory_percent‚96.79460525512695& proc.memory.availableMB‚ 525.171875 cpu‚25.259237798308487 disk.sda2.out‚0$ disk.dm-0.out‚117613.10009765625 disk.sda.out‚ 138.75390625# proc.memory.rssMB‚ 3344.33984375 proc.cpu.threads‚17 network.sent‚ 4.353211e+06 disk.dm-1.out‚ 138.75390625 disk.drbd1260.out‚ 0.65234375 disk.sda2.in‚0' disk.nvme0n1.out‚117613.10009765625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚10077.60546875* proc.memory.percent‚20.412230491638184 network.recv‚ 1.436798e+06) disk./.usagePercent‚78.47708895389341% disk./.usageGB‚1169.3653373718262 disk.dm-1.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚10077.60546875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚10077.60546875 disk.sda3.out‚ 138.75390625) disk.nvme0c0n1.out‚117613.10009765625 disk.sda1.out‚0Ís“ç‚(:ˆ ‚ÁèÔòÕ±ß gpu.0.gpu‚75.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚238.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l7l¤`è‚(:Õ ‚ÁèÔÉ­’à disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚96.79603576660156# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.259894485507047 network.sent‚ 4.358197e+06 network.recv‚ 1.44025e+06( disk./.usagePercent‚78.4754835306462% disk./.usageGB‚1169.3414154052734 disk.drbd1260.out‚0.65625 disk.sda3.out‚139.25# disk.nvme0c0n1.in‚ 10132.9921875 disk.dm-1.out‚139.25 disk.dm-0.in‚ 10132.9921875 disk.sda.out‚139.25 disk.sda1.in‚0$ proc.memory.availableMB‚524.9375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚117766.6640625! disk.nvme0n1.in‚ 10132.9921875# disk.nvme0n1.out‚117766.6640625 disk.sda3.in‚0% disk.nvme0c0n1.out‚117766.6640625|ϵ©wé‚(:ì ‘ÁèÔøº‡Þ proc.cpu.threads‚17 network.sent‚ 4.364614e+06 network.recv‚ 1.444995e+06% disk./.usageGB‚1168.1142463684082# disk.nvme0c0n1.in‚ 10132.9921875 disk.sda.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 524.67578125 disk.dm-0.in‚ 10132.9921875! disk.nvme0n1.in‚ 10132.9921875 disk.sda3.out‚ 139.671875 disk.sda.out‚ 139.671875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.65625) disk./.usagePercent‚78.39312718691876% disk.nvme0n1.out‚118287.224609375 disk.sda3.in‚0' disk.nvme0c0n1.out‚118287.224609375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚96.79763317108154# proc.memory.rssMB‚ 3344.33984375" disk.dm-0.out‚118287.224609375 disk.dm-1.out‚ 139.671875* proc.memory.percent‚20.412230491638184 cpu‚25.26015334753579¯a;‘ê‚(:† ‘ÁèÔ‚æÓÞ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚217.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾­õ42ë- _wandbruntime‚4619  _runtime‚46190 ¾ÅÇìÁ & lr-AdamW/pg2‚0.0009239398526287277 trainer/global_step‚1649 % _timestamp‚1.7884857923708959e+09 _runtime‚4619.912368189 _step‚130 & lr-AdamW/pg1‚0.0009239398526287277'ØU?ÌíÆ % _timestamp‚1.7884857923708959e+09 _runtime‚4619.912368189 _step‚130 & lr-AdamW/pg1‚0.0009239398526287277 & lr-AdamW/pg2‚0.0009239398526287277 trainer/global_step‚1649‚"4^ë2î- _wandbruntime‚4619  _runtime‚4619Ùxù'ïóO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34379641 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5011535883264 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40733251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚537259879122432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5007967053312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9189633 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40733251 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537259879122432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537256284864512 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚532248343239168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚766 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚515113663038139 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5007967053312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23280464 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102116308857856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75112892 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚532248343239168 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9189636 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚510108973006800 _runtime‚4619.915971472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 trainer/global_step‚1649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34379647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5011535883264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102116337955840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34379641 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40759334 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414068092416 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5001123194003 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40759334 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚532248317811200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6Ì¡)<ðøO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40733251 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537259879122432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537256284864512 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚532248343239168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚766 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚515113663038139 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5007967053312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23280464 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.currenty£KšÂ‚3136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102116308857856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75112892 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚532248343239168 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9189636 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚510108973006800 _runtime‚4619.915971472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 trainer/global_step‚1649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34379647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5011535883264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102116337955840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34379641 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40759334 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414068092416 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5001123194003 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40759334 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚532248317811200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34379641 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5011535883264 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40733251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚537259879122432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5007967053312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9189633 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310ƒ^AuA“ñ‚(:ˆ  ÁèÔ¹¤Þ gpu.0.gpu‚75.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚176.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏMm´lò‚(:á  ÁèÔ°®šß disk.sda.out‚ 140.0078125( proc.memory.availableMB‚ 524.93359375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 10132.9921875 disk.sda3.in‚0 network.recv‚ 1.449179e+06% disk./.usageGB‚1168.1147537231445 disk.drbd1260.out‚ 0.66015625 disk.dm-0.in‚ 10132.9921875 disk.dm-0.out‚ 118575.625# disk.nvme0c0n1.in‚ 10132.9921875 network.sent‚ 4.370437e+06) disk./.usagePercent‚78.39316123592081! disk.nvme0c0n1.out‚ 118575.625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 140.0078125$ memory_percent‚96.79605960845947 disk.sda3.out‚ 140.0078125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.260267642848916 disk.nvme0n1.out‚ 118575.625 >ø2ó- _wandbruntime‚4622  _runtime‚4622æuzæ'ôýM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23294295 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚778 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚532562934255616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5014530483200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1649 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34399897 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚532562908827648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4622.107593881 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6417935172096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚515418324548161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75183549 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882112 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40757543 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40757543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108595819003904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5004113706617 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6417842387968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚510410644004000 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5007680544161 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010961639936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102177883831808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102177854733824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚515414732282649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34399897 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5007680544161 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010961639936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102177883831808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102177854733824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚515414732282649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34399897 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23294295 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚778 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚532562934255616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5014530483200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1649 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34399897 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚532562908827648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4622.107593881 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6417935172096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚515418324548161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75183549 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121882112„‘•'2ö- _wandbruntime‚4622  _runtime‚4622Ãäà\÷Ö /loss/FRt3/detection_loss‚2.2733266039209052 0loss/SRqqt3/detection_loss‚2.177458378366935 /loss/FBt2/detection_loss‚0.7652925992178382 % _timestamp‚1.7884857945679772e+09 0loss/FRt3/assignment_loss‚3.8723716742389804 0loss/FRt4/assignment_loss‚1.9878784607327173 /loss/FBt3/detection_loss‚2.6947344426185853 epoch‚2 trainer/global_step‚1649 _runtime‚4622.109133814 /loss/FRt1/detection_loss‚3.2387490549752878 _step‚133 .loss/FRt2/detection_loss‚2.031680088029021 2loss/SRqqt1/assignment_loss‚2.1268863871974766 1loss/FBt1/assignment_loss‚0.19428963659967735 1loss/FBt4/assignment_loss‚0.12977943311181242 0loss/SRqqt1/detection_loss‚2.491222126303979 .loss/FBt4/detection_loss‚2.659453116930448 0loss/FRt2/assignment_loss‚2.5556517521540325 0loss/FBt2/assignment_loss‚0.1392129442379839 1loss/FBt3/assignment_loss‚0.15711612003900471 2loss/SRqqt2/assignment_loss‚1.7257156699503724 1loss/SRqqt2/detection_loss‚2.5345701998785923 /loss/FRt1/assignment_loss‚4.106562833453333 /loss/FRt4/detection_loss‚1.8469108396342822 0loss/SRqqt4/detection_loss‚1.033357601094095 .loss/FBt1/detection_loss‚3.001594048614303 &loss/total_loss‚46.446949446830175 2loss/SRqqt3/assignment_loss‚1.5376275413168163 2loss/SRqqt4/assignment_loss‚1.1655078942136925OÿÏDaøÛ 0loss/FBt2/assignment_loss‚0.1392129442379839 1loss/FBt3/assignment_loss‚0.15711612003900471 2loss/SRqqt2/assignment_loss‚1.7257156699503724 1loss/SRqqt2/detection_loss‚2.5345701998785923 /loss/FRt1/assignment_loss‚4.106562833453333 /loss/FRt4/detection_loss‚1.8469108396342822 0loss/SRqqt4/detection_loss‚1.033357601094095 .loss/FBt1/detection_loss‚3.001594048614303 &loss/total_loss‚46.446949446830175 2loss/SRqqt3/assignment_loss‚1.5376275413168163 2loss/SRqqt4/assignment_loss‚1.1655078942136925 /loss/FRt3/detection_loss‚2.2733266039209052 0loss/SRqqt3/detection_loss‚2.177458378366935 /loss/FBt2/detection_loss‚0.7652925992178382 % _timestamp‚1.7884857945679772e+09 0loss/FRt3/assignment_loss‚3.8723716742389804 0loss/FRt4/assignment_loss‚1.9878784607327173 /loss/FBt3/detection_loss‚2.6947344426185853 epoch‚2 trainer/global_step‚1649 _runtime‚4622.109133814 /loss/FRt1/detection_loss‚3.2387490549752878 _step‚133 .loss/FRt2/detection_loss‚2.031680088029021 2loss/SRqqt1/assignment_loss‚2.1268863871974766 1loss/FBt1/assignment_loss‚0.19428963659967735 1loss/FBt4/assignment_loss‚0.12977943311181242 0loss/SRqqt1/detection_loss‚2.491222126303979 .loss/FBt4/detection_loss‚2.659453116930448 0loss/FRt2/assignment_loss‚2.5556517521540325…¥–hÜ“ù‚(:ˆ ¯ÁèÔ›µÚß gpu.0.gpu‚96.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚230.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.0558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅâC»vú‚(:ë ¯ÁèÔúž…à disk.sda1.out‚0& proc.memory.availableMB‚ 524.890625 cpu‚25.260331783984647# disk.nvme0c0n1.in‚ 10132.9921875# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17) disk./.usagePercent‚78.40307973581989 disk.dm-0.in‚ 10132.9921875 disk.sda.in‚0 disk.sda.out‚ 140.3984375 disk.dm-1.in‚0 disk.dm-1.out‚ 140.3984375 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.454489e+06 disk.dm-0.out‚118863.2421875! disk.nvme0n1.in‚ 10132.9921875# disk.nvme0n1.out‚118863.2421875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.66015625$ memory_percent‚96.79632186889648 network.sent‚ 4.445721e+06% disk./.usageGB‚1168.2625465393066 disk.sda3.out‚ 140.3984375% disk.nvme0c0n1.out‚118863.2421875 disk.sda1.in‚0[Æa9€û‚(:õ ¾ÁèÔ™üÈà disk.nvme0n1.in‚ 10256.609375& disk.nvme0n1.out‚119039.7900390625( disk.nvme0c0n1.out‚119039.7900390625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 140.80078125 disk.drbd1260.out‚ 0.6640625) disk./.usagePercent‚78.41138334019297 disk.sda3.out‚ 140.80078125" disk.nvme0c0n1.in‚ 10256.609375$ memory_percent‚96.79481983184814* proc.memory.percent‚20.412230491638184 network.sent‚ 4.521508e+06% disk./.usageGB‚1168.3862762451172 disk.dm-0.in‚ 10256.609375 disk.sda.out‚ 140.80078125 disk.drbd1260.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 525.13671875# proc.memory.rssMB‚ 3344.33984375 network.recv‚ 1.459947e+06# disk.dm-0.out‚119039.7900390625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.260399307954874 proc.cpu.threads‚17.>Èžü‚(:“ ¾ÁèÔÐñ¶â gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.441' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.488200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤„’ý‚(:‡ ÍÁèÔ˜¨»Ý gpu.0.gpu‚28.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚312.336' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚62.4672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àQÄ€þ‚(:õ ÍÁèÔý¥Ýß# disk.dm-0.out‚119287.0830078125 disk.sda3.out‚ 141.26171875 disk.sda.in‚0 disk.sda.out‚ 141.26171875 disk.dm-1.in‚0 disk.dm-1.out‚ 141.26171875 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚119287.0830078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 10256.609375 disk.drbd1260.out‚ 0.6640625$ memory_percent‚96.79787158966064 network.recv‚ 1.464945e+06( proc.memory.availableMB‚ 524.63671875# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.260639072909385 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 10256.609375 network.sent‚ 4.527801e+06) disk./.usagePercent‚78.40384135823425 disk.dm-0.in‚ 10256.609375& disk.nvme0n1.out‚119287.0830078125 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1168.27389526367192FÎujÿ‚(:ˆ ÜÁèÔ¥•§à gpu.0.gpu‚97.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚68.115310668Á«‚r)94531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚49.0 gpu.0.powerWatts‚331.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆà·€€‚(:õ ÜÁèÔÏÄæ disk.drbd1260.out‚ 0.66796875) disk./.usagePercent‚78.41146398256626% disk./.usageGB‚1168.3874778747559 disk.dm-0.in‚ 10256.609375 disk.nvme0n1.in‚ 10256.609375 disk.sda.out‚ 141.6484375 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚96.79481983184814 disk.sda3.in‚0 disk.sda1.out‚0 cpu‚25.26085102994122 proc.cpu.threads‚17 network.sent‚ 4.533676e+06 network.recv‚ 1.469259e+06 disk.dm-1.out‚ 141.6484375 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 525.13671875# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184$ disk.dm-0.out‚119647.07470703125' disk.nvme0n1.out‚119647.07470703125 disk.sda3.out‚ 141.6484375" disk.nvme0c0n1.in‚ 10256.609375) disk.nvme0c0n1.out‚119647.07470703125 disk.sda.in‚0 disk.drbd1260.in‚0× -u‚(:ê ëÁèÔ©¹àà disk.dm-1.in‚0 disk.dm-1.out‚142.0625 disk.sda2.in‚0% proc.memory.availableMB‚ 524.59375' disk.nvme0n1.out‚120120.89501953125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 10256.609375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.66796875# proc.memory.rssMB‚ 3344.33984375 cpu‚25.261085207878104 proc.cpu.threads‚17 network.recv‚ 1.472859e+06% disk./.usageGB‚1168.4440307617188 disk.sda3.out‚142.0625 disk.sda.in‚0 disk.sda.out‚142.0625$ memory_percent‚96.79813385009766 network.sent‚ 4.540796e+06$ disk.dm-0.out‚120120.89501953125 disk.nvme0n1.in‚ 10256.609375) disk.nvme0c0n1.out‚120120.89501953125 disk.drbd1260.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.412230491638184) disk./.usagePercent‚78.41525929426136 disk.dm-0.in‚ 10256.609375 disk.sda1.in‚0b˜å“‚‚(:ˆ ëÁèÔõ¦×à gpu.0.gpu‚67.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚180.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íW¥lƒ‚(:á úÁèÔ´€áà disk.sda3.out‚ 142.53125% proc.memory.availableMB‚ 524.84375 network.recv‚ 1.475891e+06 disk.sda2.in‚0# memory_percent‚96.7966079711914* proc.memory.percent‚20.412230491638184 cpu‚25.261207224574523 proc.cpu.threads‚17% disk./.usageGB‚1168.4520072937012# disk.nvme0c0n1.in‚ 10381.8984375 disk.sda.out‚ 142.53125 disk.dm-1.out‚ 142.53125 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3344.33984375 network.sent‚ 4.543785e+06% disk.nvme0c0n1.out‚120417.2421875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.671875 disk.dm-0.in‚ 10381.8984375 disk.dm-0.out‚120417.2421875) disk./.usagePercent‚78.41579460601547 disk.sda.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 10381.8984375# disk.nvme0n1.out‚120417.2421875 disk.sda3.in‚0¨oŠ‘„‚(:† úÁèÔ“îëá gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚229.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.954 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰´UÊi…‚(:Þ ‰ÂèÔË©½ß disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚20.412230491638184 cpu‚25.261550519035765 disk.dm-0.in‚ 10381.8984375# disk.nvme0c0n1.in‚ 10381.8984375 disk.sda.out‚ 143.12890625 proc.cpu.threads‚17 disk.sda3.in‚0 network.recv‚ 1.480732e+06 disk.drbd1260.in‚0 disk.dm-0.out‚ 120494.25 disk.nvme0n1.out‚ 120494.25 disk.sda.in‚0 disk.dm-1.out‚ 143.12890625( proc.memory.availableMB‚ 525.08984375 network.sent‚ 4.551126e+06 disk.sda2.in‚0! disk.nvme0n1.in‚ 10381.8984375 disk.sda3.out‚ 143.12890625 disk.nvme0c0n1.out‚ 120494.25 disk.sda1.in‚0$ memory_percent‚96.79510593414307# proc.memory.rssMB‚ 3344.33984375) disk./.usagePercent‚78.41607979840862$ disk./.usageGB‚1168.456256866455 disk.drbd1260.out‚0.671875 disk.sda2.out‚0α`†‚(:’ ‰ÂèÔ¹©Ÿä gpu.0.gpu‚40.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚164.831' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚32.96619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ùŒ 2‡- _wandbruntime‚4739  _runtime‚4739;á•>Ĉ¾ % lr-AdamW/pg2‚0.000915450584098632 trainer/global_step‚1699 $ _timestamp‚1.788485911780524e+09 _runtime‚4739.322122846 _step‚134 % lr-AdamW/pg1‚0.000915450584098632wQ©É‰Ã % lr-AdamW/pg1‚0.000915450584098632 % lr-AdamW/pg2‚0.000915450584098632 trainer/global_step‚1699 $ _timestamp‚1.788485911780524e+09 _runtime‚4739.322122846 _step‚134†Þ Àu2Š- _wandbruntime‚4739  _runtime‚4739È&_ù'‹óO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14505694 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚530346765700847 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 _step‚135 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23970302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6606975804928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23969527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553139187104256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35392147 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35392147 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5157724212736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9463833 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5157724212736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚547977894061568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77369433 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111800700356608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553135592846336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚547977894061568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5161293042688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6607068602368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105193631754240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5154242834047 _runtime‚4739.324210662 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41977286 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9463836 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41951203 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41951203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14506466 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚547977868633600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚530343173435543 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚547977868633600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 trainer/global_step‚1699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448镤þ'ŒøO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111800700356608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553135592846336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚547977894061568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5161293042688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6607068602368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105193631754240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5154242834047 _runtime‚4739.324210662 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41977286 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9463836 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41951203 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41951203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14506466 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92797440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121895424 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚547977868633600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚530343173435543 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚547977868633600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 trainer/global_step‚1699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14505694 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚530346765700847 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 _step‚135 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23970302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6606975804928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23969527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553139187104256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35392147 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35392147 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5157724212736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9463833 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5157724212736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚547977894061568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77369433 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0‡tõ÷Ξ‚(:“ ˜ÂèÔÂŽÐÝ gpu.0.gpu‚97.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚45.0 gpu.0.powerWatts‚120.097' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.019399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùkuÏ„Ž‚(:ù ˜ÂèÔ¿ãÒß* proc.memory.percent‚20.412230491638184 cpu‚25.26177242888983 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 10381.8984375 disk.sda2.in‚0! disk.nvme0n1.in‚ 10381.8984375 disk.sda3.in‚0$ memory_percent‚96.79665565490723 disk.dm-1.out‚ 143.48828125 disk.drbd1260.out‚ 0.67578125 network.sent‚ 4.555111e+06( disk./.usagePercent‚78.4144984398696% disk./.usageGB‚1168.4326934814453) disk.nvme0c0n1.out‚120589.87939453125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 143.48828125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 10381.8984375$ disk.dm-0.out‚120589.87939453125' disk.nvme0n1.out‚120589.87939453125' proc.memory.availableMB‚ 524.8359375# proc.memory.rssMB‚ 3344.33984375 network.recv‚ 1.485155e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 143.48828125]ÚÍ2- _wandbruntime‚4741  _runtime‚4741Ã7Ì4’ýM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9469320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35412397 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5160718799360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14514704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14513966 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117687808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77414001 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111866012976640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77414001 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 UEDeviceStaÀлqtsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5160718799360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚525494168436032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23983283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚530647834945357 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 _step‚136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚741 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚530651427210869 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77387886 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 trainer/global_step‚1699 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105255177630208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9469317 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553453178449408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6610864444416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23984024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553456772720640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77387886 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105255148532224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35412391 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41975495 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35412391 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35412397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553456772720640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41975495 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 FŒÞ5'‘‚N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77387886 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 trainer/global_step‚1699 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105255177630208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9469317 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553453178449408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6610864444416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23984024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553456772720640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77387886 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105255148532224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35412391 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41975495 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35412391 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35412397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553456772720640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41975495 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35412397 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5160718799360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14514704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14513966 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117687808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77414001 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111866012976640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77414001 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5160718799360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚525494168436032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23983283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚530647834945357 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 _step‚136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚741 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚530651427210869ˆ¢ò>ù2’- _wandbruntime‚4741  _runtime‚4741µ9v(Z“Ô 0loss/SRqqt4/detection_loss‚0.901464605011046 /loss/FBt1/detection_loss‚2.3639549146860075 2loss/SRqqt2/assignment_loss‚1.8280947491890047 2loss/SRqqt4/assignment_loss‚0.9910218237265944 /loss/FRt3/detection_loss‚2.2156509821151578 .loss/SRqqt2/detection_loss‚2.3325010341842 /loss/FBt2/detection_loss‚0.7381326432993275 .loss/FBt3/detection_loss‚2.326651289127767 epoch‚2 0loss/FRt3/assignment_loss‚3.5640980233979778 2loss/SRqqt3/assignment_loss‚1.3999801639219125 1loss/FBt2/assignment_loss‚0.33072294963259913 /loss/FBt4/detection_loss‚2.4512389646126675 trainer/global_step‚1699 _runtime‚4741.676140421 1loss/FBt1/assignment_loss‚0.24535914087514007 %loss/total_loss‚43.90059756169555 1loss/FBt3/assignment_loss‚0.24497144144788763 .loss/FRt1/detection_loss‚3.373351822065753 /loss/FRt2/detection_loss‚2.0434063698323266 1loss/SRqqt1/detection_loss‚2.7045245942744343 1loss/SRqqt3/detection_loss‚2.2161909721295037 _step‚137 /loss/FRt1/assignment_loss‚3.878903721654138 2loss/SRqqt1/assignment_loss‚1.9565787454220382 /loss/FRt4/detection_loss‚1.7676713824272157 % _timestamp‚1.7884859141347432e+09 0loss/FRt2/assignment_loss‚2.4810032518404834 .loss/FRt4/assignment_loss‚1.43652250623703 1loss/FBt4/assignment_loss‚0.10860147058533934­\‘_”Ù /loss/FRt3/detection_loss‚2.2156509821151578 .loss/SRqqt2/detection_loss‚2.3325010341842 /loss/FBt2/detection_loss‚0.7381326432993275 .loss/FBt3/detection_loss‚2.326651289127767 epoch‚2 0loss/FRt3/assignment_loss‚3.5640980233979778 2loss/SRqqt3/assignment_loss‚1.3999801639219125 1loss/FBt2/assignment_loss‚0.33072294963259913 /loss/FBt4/detection_loss‚2.4512389646126675 trainer/global_step‚1699 _runtime‚4741.676140421 1loss/FBt1/assignment_loss‚0.24535914087514007 %loss/total_loss‚43.90059756169555 1loss/FBt3/assignment_loss‚0.24497144144788763 .loss/FRt1/detection_loss‚3.373351822065753 /loss/FRt2/detection_loss‚2.0434063698323266 1loss/SRqqt1/detection_loss‚2.7045245942744343 1loss/SRqqt3/detection_loss‚2.2161909721295037 _step‚137 /loss/FRt1/assignment_loss‚3.878903721654138 2loss/SRqqt1/assignment_loss‚1.9565787454220382 /loss/FRt4/detection_loss‚1.7676713824272157 % _timestamp‚1.7884859141347432e+09 0loss/FRt2/assignment_loss‚2.4810032518404834 .loss/FRt4/assignment_loss‚1.43652250623703 1loss/FBt4/assignment_loss‚0.10860147058533934 0loss/SRqqt4/detection_loss‚0.901464605011046 /loss/FBt1/detection_loss‚2.3639549146860075 2loss/SRqqt2/assignment_loss‚1.8280947491890047 2loss/SRqqt4/assignment_loss‚0.9910218237265944‰”-­ž•‚(:“ §ÂèÔ­ž®ß gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚246.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆñVt|–‚(:ñ §ÂèÔÕ·ºà disk.dm-1.in‚0$ disk.dm-0.out‚120853.54345703125! disk.nvme0n1.in‚ 10381.8984375' disk.nvme0n1.out‚120853.54345703125' proc.memory.availableMB‚ 525.0390625# proc.memory.rssMB‚ 3344.33984375 proc.cpu.threads‚17 network.recv‚ 1.49027e+06 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 network.sent‚ 4.643235e+06 disk.sda3.out‚ 143.921875 disk.sda.in‚0 disk.sda.out‚ 143.921875 disk.drbd1260.out‚ 0.67578125 disk.sda2.in‚0* proc.memory.percent‚20.412230491638184 cpu‚25.26188219668008) disk./.usagePercent‚78.42455825592758# disk.nvme0c0n1.in‚ 10381.8984375) disk.nvme0c0n1.out‚120853.54345703125 disk.sda1.out‚0 disk.dm-1.out‚ 143.921875 disk.sda2.out‚0 disk.dm-0.in‚ 10381.8984375# memory_percent‚96.7954158782959$ disk./.usageGB‚1168.5825920104989lk—‚(:à ¶ÂèÔð´æà* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 network.sent‚ 4.706512e+06 network.recv‚ 1.495665e+06) disk./.usagePercent‚78.42871428223843 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚121485.123046875 disk.sda3.in‚0 disk.nvme0c0n1.in‚10504.5" disk.dm-0.out‚121485.123046875( proc.memory.availableMB‚ 525.03515625 cpu‚25.26209985072207% disk./.usageGB‚1168.6445198059082 disk.sda.out‚ 144.4140625 disk.sda1.out‚0 disk.dm-1.out‚ 144.4140625 disk.drbd1260.out‚ 0.68359375 disk.dm-0.in‚10504.5 disk.nvme0n1.in‚10504.5$ memory_percent‚96.79543972015381' disk.nvme0c0n1.out‚121485.123046875 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 144.4140625# proc.memory.rssMB‚ 3344.33984375fjß“˜‚(:ˆ ¶ÂèÔÖ¿óà gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚211.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.3638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âSÛ‘o™‚(:ä ÅÂèÔÞŽžà network.recv‚ 1.500288e+06 disk.sda.in‚0 disk.sda.out‚ 144.79296875 disk.drbd1260.in‚0 disk.dm-0.in‚10504.5 proc.cpu.threads‚17) disk./.usagePercent‚78.44129756456142% disk./.usageGB‚1168.8320198059082' disk.nvme0c0n1.out‚121924.009765625% disk.nvme0n1.out‚121924.009765625 disk.sda3.out‚ 144.79296875* proc.memory.percent‚20.412230491638184 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0" disk.dm-0.out‚121924.009765625 disk.nvme0n1.in‚10504.5 disk.sda3.in‚0$ memory_percent‚96.79543972015381( proc.memory.availableMB‚ 525.03515625 network.sent‚ 4.712859e+06 disk.nvme0c0n1.in‚10504.5 disk.sda1.in‚0 disk.dm-1.out‚ 144.79296875 disk.drbd1260.out‚ 0.68359375 disk.sda2.out‚0# proc.memory.rssMB‚ 3344.33984375 cpu‚25.262480124607958ë,ÖÑœš‚(:‘ ÅÂèÔãÑá gpu.0.gpu‚31.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚311.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚62.255999999999986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:0Ïj›‚(:ß ÔÂèԈϭâ disk.sda2.out‚0$ disk.dm-0.out‚122027.31201171875% disk./.usageGB‚1169.2080459594727) disk.nvme0c0n1.out‚122027.31201171875 disk.sda.out‚ 145.234375 disk.drbd1260.out‚0.6875# proc.memory.rssMB‚ 3344.33984375 network.sent‚ 4.718801e+06 disk.sda1.in‚0 disk.dm-0.in‚10504.5 disk.nvme0n1.in‚10504.5' disk.nvme0n1.out‚122027.31201171875( proc.memory.availableMB‚ 524.78515625* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 145.234375 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚96.79696559906006 cpu‚25.26262459995799 network.recv‚ 1.504656e+06) disk./.usagePercent‚78.46653299523409 disk.nvme0c0n1.in‚10504.5 disk.dm-1.in‚0 disk.dm-1.out‚ 145.234375 disk.sda2.in‚0Ü_ 2’œ‚(:‡ ÔÂèÔíůâ gpu.0.gpu‚95.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚247.175' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜZ.m‚(:â ãÂèÔÿû…à disk.sda2.out‚0* proc.memory.percent‚20.412230491638184% disk./.usageGB‚1169.2070274353027' disk.nvme0n1.out‚122219.85498046875) disk.nvme0c0n1.out‚122219.85498046875 disk.drbd1260.out‚0.6875# proc.memory.rssMB‚ 3344.33984375 network.sent‚ 4.723938e+06 disk.dm-0.in‚10504.5$ disk.dm-0.out‚122219.85498046875 disk.dm-1.in‚0 disk.drbd1260.in‚0 cpu‚25.26281478728138 proc.cpu.threads‚17) disk./.usagePercent‚78.46646464122244 disk.nvme0n1.in‚10504.5 disk.nvme0c0n1.in‚10504.5 disk.sda.out‚ 145.6796875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.79605960845947( proc.memory.availableMB‚ 524.93359375 network.recv‚ 1.508353e+06 disk.sda3.in‚0 disk.sda3.out‚ 145.6796875 disk.sda.in‚0 disk.dm-1.out‚ 145.6796875 disk.sda2.in‚0rÓ{žž‚(:“ ãÂèÔ¤¿’â gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚212.878' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.575599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!¿mh{Ÿ‚(:ð òÂèÔ±ŒâÝ disk.drbd1260.out‚ 0.69140625 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 146.078125( proc.memory.availableMB‚ 524.69140625 disk.sda3.in‚0 disk.sda3.out‚ 146.078125# disk.nvme0c0n1.in‚ 10662.3359375 disk.dm-1.in‚0# memory_percent‚96.7975378036499# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 proc.cpu.threads‚17 disk.sda2.out‚0# disk.dm-0.out‚122514.6787109375! disk.nvme0n1.in‚ 10662.3359375 disk.sda1.out‚0 cpu‚25.263054351968215 network.sent‚ 4.7291e+06) disk./.usagePercent‚78.48173446660974 disk.drbd1260.in‚0 disk.dm-0.in‚ 10662.3359375& disk.nvme0n1.out‚122514.6787109375( disk.nvme0c0n1.out‚122514.6787109375 disk.sda.out‚ 146.078125 network.recv‚ 1.512065e+06% disk./.usageGB‚1169.4345588684082TÞГ ‚(:ˆ òÂèÔí­‚„ gpu.0.gpu‚60.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚42.0 gpu.0.powerWatts‚117.018' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?j¡‚(:ö ÃèÔÕáÆë network.sent‚ 4.735183e+06) disk./.usagePercent‚78.48324107094874 disk.sda.out‚ 146.73046875& disk.nvme0n1.out‚122976.6201171875# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.26332833915619 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69140625 disk.dm-0.in‚ 10662.3359375( proc.memory.availableMB‚ 524.92578125% disk./.usageGB‚1169.4570083618164 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.dm-0.out‚122976.6201171875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 10662.3359375( disk.nvme0c0n1.out‚122976.6201171875 network.recv‚ 1.51671e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 146.73046875 disk.sda2.out‚0! disk.nvme0n1.in‚ 10662.3359375 disk.sda3.out‚ 146.73046875# memory_percent‚96.7961072921753‰ÍÊd“¢‚(:ˆ ÃèÔµ’í gpu.0.gpu‚62.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚327.018' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.4036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U ÐÆ2£- _wandbruntime‚4859  _runtime‚4859míƤÀ $ _timestamp‚1.788486031616129e+09 _runtime‚4859.157625039 _step‚138 & lr-AdamW/pg1‚0.0009066201148292533 & lr-AdamW/pg2‚0.0009066201148292533 trainer/global_step‚1749hÔ\û˥Š& lr-AdamW/pg1‚0.0009066201148292533 & lr-AdamW/pg2‚0.0009066201148292533 trainer/global_step‚1749 $ _timestamp‚1.788486031616129e+09 _runtime‚4859.157625039 _step‚138ŠÉ•{2¦- _wandbruntime‚4859  _runtime‚4859v6#1³§òO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79573796 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79599885 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6800072877568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545576276098251 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5307481372160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚785 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4859.161095012 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563707419456000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6799977982976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9738036 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94894592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108270925552640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 _step‚139 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36404641 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24658947 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14921696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43195238 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚782 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5311050202112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚569014900828160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5311050202112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 trainer/global_step‚1749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36404647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚569018495086080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563707444883968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24659732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43195238 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36404647 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54027607272Ñá_jE6800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14920914 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115070874437632 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚563707419456000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300228799419 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563707444883968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108270896454656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5307481372160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43169155߃”>ý'¨÷O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5311050202112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚569014900828160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5311050202112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 trainer/global_step‚1749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36404647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚569018495086080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563707444883968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24659732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43195238 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36404647 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚540276072726800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14920914 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115070874437632 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚563707419456000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300228799419 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563707444883968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108270896454656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5307481372160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43169155 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79573796 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79599885 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6800072877568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545576276098251 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5307481372160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚785 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4859.161095012 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563707419456000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6799977982976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9738036 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94894592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108270925552640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 _step‚139 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36404641 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24658947 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14921696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43195238 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚782 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448‹¯ûkž©‚(:“ ÃèÔ«¨ýÜ gpu.0.gpu‚57.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚250.926' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.185199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆý·q‚ª‚(:÷ ÃèԪ͚à# disk.nvme0c0n1.in‚ 10662.3359375) disk.nvme0c0n1.out‚123320.59521484375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.6953125 disk.dm-0.in‚ 10662.3359375$ disk.dm-0.out‚123320.59521484375) disk./.usagePercent‚78.51535696011165 disk.sda.in‚0 disk.sda.out‚ 147.29296875 disk.sda1.out‚0 disk.dm-1.out‚ 147.29296875 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 10662.3359375 network.recv‚ 1.521131e+06' disk.nvme0n1.out‚123320.59521484375& proc.memory.availableMB‚ 524.921875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 147.29296875 disk.dm-1.in‚0 disk.drbd1260.in‚0# memory_percent‚96.7961311340332# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.26355676829611 network.sent‚ 4.754124e+06% disk./.usageGB‚1169.9355583190918ëî¤2«- _wandbruntime‚4861  _runtime‚4861ã/„'¬ýM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545880937608065 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36424891 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 trainer/global_step‚1749 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24673486 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108332471428608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14929189 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 _step‚140 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36424891 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5310475958784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115136414142464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545884529873577 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79618338 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43193447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24672706 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚540577743724000 % _timestamp‚1.7884860339354699e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚564022035900416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115136292260352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6803942713856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43193447 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚569336080702464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9743520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚564022035900416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36424897 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108332442330624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 _runtime‚4861.478369716 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36424897 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5314044802048 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6803849929728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79644453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚540577718296032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5303219312033 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚569332486431232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544áNôÂ8­‚N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚540577718296032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5303219312033 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚569332486431232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545880937608065 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36424891 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 trainer/global_step‚1749 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24673486 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108332471428608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14929189 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 _step‚140 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36424891 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5310475958784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115136414142464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545884529873577 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79618338 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43193447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24672706 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚540577743724000 % _timestamp‚1.7884860339354699e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚564022035900416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115136292260352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6803942713856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43193447 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚569336080702464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9743520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚564022035900416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batEóˆ9Ð ch_end/allocation.large_pool.allocated‚36424897 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108332442330624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 _runtime‚4861.478369716 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36424897 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5314044802048 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6803849929728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79644453Œww¯{2®- _wandbruntime‚4861  _runtime‚4861¸ÒšfW¯Ñ 0loss/FBt4/assignment_loss‚0.8166043056262424 % _timestamp‚1.7884860339393313e+09 0loss/FRt2/assignment_loss‚2.7052579134702683 1loss/FBt2/assignment_loss‚0.25746661388221226 0loss/SRqqt2/detection_loss‚2.763188027102372 .loss/FBt1/detection_loss‚2.474326272805532 0loss/FBt3/assignment_loss‚0.4231966681067999 -loss/FRt2/detection_loss‚2.30446534845978 .loss/FBt4/detection_loss‚2.069619804620743 1loss/SRqqt4/detection_loss‚1.0272548449683492 /loss/FBt2/detection_loss‚0.9222171889956702 0loss/FBt1/assignment_loss‚0.2552064131593387 1loss/SRqqt1/detection_loss‚2.4767069021860757 epoch‚2 2loss/SRqqt3/assignment_loss‚1.6029082612298868 2loss/SRqqt4/assignment_loss‚0.9595351880613588 /loss/FRt1/detection_loss‚3.1031094697805552 .loss/FRt4/detection_loss‚1.558664883936749 0loss/FRt1/assignment_loss‚3.7808083593845367 1loss/SRqqt1/assignment_loss‚1.412191771263001 /loss/FRt3/detection_loss‚2.1386265984603336 .loss/FBt3/detection_loss‚2.338383980668508 &loss/total_loss‚44.968626533101606 _step‚141 0loss/FRt4/assignment_loss‚1.6127396878742037 1loss/SRqqt2/assignment_loss‚1.995750516653061 0loss/SRqqt3/detection_loss‚2.146244223065236 trainer/global_step‚1749 _runtime‚4861.480667459 0loss/FRt3/assignment_loss‚3.8241532893407912âòÉ\°Ö 0loss/FRt2/assignment_loss‚2.7052579134702683 1loss/FBt2/assignment_loss‚0.25746661388221226 0loss/SRqqt2/detection_loss‚2.763188027102372 .loss/FBt1/detection_loss‚2.474326272805532 0loss/FBt3/assignment_loss‚0.4231966681067999 -loss/FRt2/detection_loss‚2.30446534845978 .loss/FBt4/detection_loss‚2.069619804620743 1loss/SRqqt4/detection_loss‚1.0272548449683492 /loss/FBt2/detection_loss‚0.9222171889956702 0loss/FBt1/assignment_loss‚0.2552064131593387 1loss/SRqqt1/detection_loss‚2.4767069021860757 epoch‚2 2loss/SRqqt3/assignment_loss‚1.6029082612298868 2loss/SRqqt4/assignment_loss‚0.9595351880613588 /loss/FRt1/detection_loss‚3.1031094697805552 .loss/FRt4/detection_loss‚1.558664883936749 0loss/FRt1/assignment_loss‚3.7808083593845367 1loss/SRqqt1/assignment_loss‚1.412191771263001 /loss/FRt3/detection_loss‚2.1386265984603336 .loss/FBt3/detection_loss‚2.338383980668508 &loss/total_loss‚44.968626533101606 _step‚141 0loss/FRt4/assignment_loss‚1.6127396878742037 1loss/SRqqt2/assignment_loss‚1.995750516653061 0loss/SRqqt3/detection_loss‚2.146244223065236 trainer/global_step‚1749 _runtime‚4861.480667459 0loss/FRt3/assignment_loss‚3.8241532893407912 0loss/FBt4/assignment_loss‚0.8166043056262424 % _timestamp‚1.7884860339393313e+09È›§’±‚(:‡ ŸÃèÔãæ‘Ý gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚244.435' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.887 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f¼§²‚(:ö ŸÃèÔæ‰à network.recv‚ 1.526612e+06% disk./.usageGB‚1169.9570083618164) disk.nvme0c0n1.out‚123759.48974609375 disk.sda1.in‚0 disk.dm-1.out‚ 147.8984375 disk.drbd1260.out‚ 0.6953125 disk.sda2.out‚0$ disk.dm-0.out‚123759.48974609375) disk./.usagePercent‚78.51679649047668# disk.nvme0c0n1.in‚ 10662.3359375 disk.sda.out‚ 147.8984375 disk.sda1.out‚0' disk.nvme0n1.out‚123759.48974609375 disk.sda3.in‚0# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 network.sent‚ 4.816355e+06 disk.sda.in‚0 disk.dm-0.in‚ 10662.3359375 disk.sda3.out‚ 147.8984375$ memory_percent‚96.79651260375977& proc.memory.availableMB‚ 524.859375 cpu‚25.263719198864212 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 10662.3359375¼b<†³‚(:û ®ÃèÔîÍß disk.dm-1.in‚0 disk.drbd1260.in‚0 network.recv‚ 1.53162e+06) disk./.usagePercent‚78.51680468271778) disk.nvme0c0n1.out‚124054.40185546875 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚10723.47265625# disk.nvme0c0n1.in‚ 10723.4453125 disk.sda1.out‚0$ disk.dm-0.out‚124054.40185546875' disk.nvme0n1.out‚124054.40185546875$ memory_percent‚96.79503440856934# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.263889394460836 disk.drbd1260.out‚ 0.69921875 network.sent‚ 4.894247e+06$ disk./.usageGB‚1169.957130432129 disk.sda.out‚ 148.21484375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 148.21484375 disk.dm-1.out‚ 148.21484375 disk.sda2.out‚0" disk.nvme0n1.in‚10723.47265625' proc.memory.availableMB‚ 525.1015625 proc.cpu.threads‚17#W—Vž´‚(:“ ®ÃèÔ ×»â gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚43.0 gpu.0.powerWatts‚147.917' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.583399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶V…‚µ‚(:÷ ½ÃèÔ¦ÙÖá disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚124108.80029296875' disk.nvme0n1.out‚124108.80029296875 proc.cpu.threads‚17 network.sent‚ 4.898569e+06 network.recv‚ 1.536338e+06) disk./.usagePercent‚78.51680340268011) disk.nvme0c0n1.out‚124108.80029296875 disk.sda.out‚ 148.7265625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69921875 disk.sda3.out‚ 148.7265625# disk.nvme0c0n1.in‚ 10779.3203125 disk.dm-1.out‚ 148.7265625 disk.dm-0.in‚ 10779.3203125$ memory_percent‚96.79653644561768* proc.memory.percent‚20.412230491638184 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 10779.3203125( proc.memory.availableMB‚ 524.85546875# proc.memory.rssMB‚ 3344.33984375 cpu‚25.2641220156578% disk./.usageGB‚1169.9571113586426¦¯º¼ž¶‚(:“ ½ÃèÔÀ­Œã gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚241.514' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.302800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üñ°;·‚(:ô ÌÃèÔ»ÝùÞ disk.sda.out‚ 149.1328125 disk.sda1.out‚0# disk.dm-0.out‚124454.8095703125# disk.nvme0c0n1.in‚ 10779.3203125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 10779.3203125! disk.nvme0n1.in‚ 10779.3203125# proc.memory.rssMB‚ 3344.33984375 cpu‚25.26435818441326) disk./.usagePercent‚78.54878386385741 disk.drbd1260.out‚ 0.70703125 disk.sda2.in‚0& disk.nvme0n1.out‚124468.0595703125 disk.sda3.in‚0 disk.sda3.out‚ 149.1328125' proc.memory.availableMB‚ 524.1015625* proc.memory.percent‚20.412230491638184% disk./.usageGB‚1170.4336433410645 disk.dm-1.in‚0 disk.dm-1.out‚ 149.1328125 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚124468.0595703125$ memory_percent‚96.80113792419434 network.sent‚ 4.906424e+06 network.recv‚ 1.540652e+06ZUÀô¸‚(:’ ÌÃèÔ©ƒ¨ß gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚206.723' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚41.34460000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í@„¹‚(:ù ÛÃèÔ„´êà& disk.nvme0n1.out‚124931.8173828125 disk.sda3.in‚0( disk.nvme0c0n1.out‚124931.8173828125# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 cpu‚25.264595054203134 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 10779.3203125# disk.dm-0.out‚124931.8173828125! disk.nvme0n1.in‚ 10779.3203125 disk.sda1.in‚0 disk.dm-1.out‚ 149.51171875$ memory_percent‚96.79510593414307 network.sent‚ 4.911536e+06 network.recv‚ 1.544284e+06 disk.sda3.out‚ 149.51171875# disk.nvme0c0n1.in‚ 10779.3203125 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.70703125) disk./.usagePercent‚78.55035882220805 disk.sda.out‚ 149.51171875 disk.sda1.out‚0( proc.memory.availableMB‚ 525.08984375 proc.cpu.threads‚17% disk./.usageGB‚1170.4571113586426®Æ™-“º‚(:ˆ ÛÃèÔ’Í…â gpu.0.gpu‚67.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚49.0 gpu.0.powerWatts‚325.144' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.0288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€éž»‚(:“ êÃèÔžÓùÝ gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚241.795' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.358999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ’F…¼‚(:ú êÃèÔšýöÞ disk.dm-1.in‚0$ memory_percent‚96.79663181304932 network.recv‚ 1.547893e+06 disk.drbd1260.in‚0 cpu‚25.264770082169168 proc.cpu.threads‚17( disk./.usagePercent‚78.5487546789985% disk./.usageGB‚1170.4332084655762 disk.drbd1260.out‚ 0.7109375 disk.dm-0.in‚ 10779.3203125$ disk.dm-0.out‚125371.97900390625# disk.nvme0c0n1.in‚ 10779.3203125) disk.nvme0c0n1.out‚125371.97900390625 disk.sda.in‚0 network.sent‚ 4.914733e+06! disk.nvme0n1.in‚ 10779.3203125 disk.sda3.in‚0 disk.sda.out‚ 149.83984375 disk.dm-1.out‚ 149.83984375( proc.memory.availableMB‚ 524.83984375# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚125371.97900390625 disk.sda3.out‚ 149.83984375 disk.sda1.in‚0 disk.sda1.out‚0c’ŽÝˆ½‚(:ý ùÃèÔÆ‹à% disk./.usageGB‚1170.4566764831543 disk.sda.out‚ 150.44140625 disk.sda2.in‚0$ memory_percent‚96.80123329162598' proc.memory.availableMB‚ 524.0859375# proc.memory.rssMB‚ 3344.33984375* proc.memory.percent‚20.412230491638184 network.sent‚ 4.922974e+06) disk./.usagePercent‚78.55032963734915$ disk.nvme0c0n1.in‚10932.62890625 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚10932.62890625$ disk.dm-0.out‚125595.63525390625 network.recv‚ 1.552631e+06) disk.nvme0c0n1.out‚125595.63525390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 150.44140625 disk.sda2.out‚0' disk.nvme0n1.out‚125595.63525390625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.7109375" disk.nvme0n1.in‚10932.62890625 disk.sda3.out‚ 150.44140625 cpu‚25.264912791078224 proc.cpu.threads‚17{Qú÷“¾‚(:ˆ ùÃèÔ‡úéä gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ#‘2¿- _wandbruntime‚4977  _runtime‚4977öL^’ÆÀÀ _runtime‚ 4977.74208557 _step‚142 & lr-AdamW/pg1‚0.0008974558176540163 & lr-AdamW/pg2‚0.0008974558176540163 trainer/global_step‚1799 % _timestamp‚1.7884861502005653e+09¤rËÁÅ & lr-AdamW/pg1‚0.0008974558176540163 & lr-AdamW/pg2‚0.0008974558176540163 trainer/global_step‚1799 % _timestamp‚1.7884861502005653e+09 _runtime‚ 4977.74208557 _step‚142Ž¢í²’2Â- _wandbruntime‚4977  _runtime‚4977\b)‹ú'ÃôO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460807361536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584897803067904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5453348439463 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37417141 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15335811 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 _step‚143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579436995706368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579436970278400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555359622586800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6992912356864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 4977.74349614 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5449781602127 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37417147 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584894208809984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5457238531584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6993007251456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579436970278400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884861502019126e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44413190 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584894208809984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118341102609920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44413190 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5457238531584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25348834 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81804248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111348219351040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118341226602496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37417141 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44387107 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123992576 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚111348190253056 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584897803067904 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81830337 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15336598 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 trainer/global_step‚1799 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460807361536 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚560812971026263 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44387107 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579436995706368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25348044 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10012233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555359597158832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94894592 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37417147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚560809378760959 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10012236 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0C_ìÄùO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6992912356864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 4977.74349614 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5449781602127 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37417147 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584894208809984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5457238531584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568829952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6993007251456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579436970278400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884861502019126e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44413190 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584894208809984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118341102609920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44413190 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5457238531584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25348834 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81804248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111348219351040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594257920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118341226602496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37417141 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44387107 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123992576 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚111348190253056 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584897803067904 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81830337 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568829952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15336598 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 trainer/global_step‚1799 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460807361536 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceSta°âtëtsMonitor.on_train_batch_start/requested_bytes.all.allocated‚560812971026263 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44387107 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579436995706368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25348044 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10012233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555359597158832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94894592 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37417147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚560809378760959 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10012236 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460807361536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584897803067904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5453348439463 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594257920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37417141 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15335811 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 _step‚143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579436995706368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579436970278400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555359622586800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 533976842249J˜v2Å- _wandbruntime‚4979  _runtime‚4979C¡'ÆþM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5460233118208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884861524138427e+09 _runtime‚ 4979.95603042 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81848790 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44411399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118406492985344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25361817 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6996756856320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5460233118208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37437397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6996851737600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81874905 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111409736129024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81874905 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585215388684288 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5463801961472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5456338952285 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579751586722816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579751561294848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44437508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15344100 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555661268156032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555661293584000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37437397 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585211794413056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81848790 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37437391 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5463801961472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15344890 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44411399 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585215388684288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561114040270773 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561117632536285 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136ÂÖ® 'ǃN PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81874905 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111409736129024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81874905 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585215388684288 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903040 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5463801961472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5456338952285 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579751586722816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579751561294848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44437508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15344100 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555661268156032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555661293584000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37437397 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585211794413056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81848790 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37437391 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5463801961472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15344890 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44411399 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585215388684288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561114040270773 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561117632536285 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5460233118208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884861524138427e+09 _runtime‚ 4979.95603042 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81848790 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44411399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123979264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452084 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118406492985344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25361817 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6996756856320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5460233118208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37437397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6996851737600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9772727489471436 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 particle/accuracy_2_of_3‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt2FBt/event_proportion‚0.008757560513913631 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.21131975948810577 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4451747536659241 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.2586805522441864 particle/accuracy_1_of_3‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.27454856038093567 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19466783106327057 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.28552383184432983 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23497183620929718 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 +jet/accuracy_3_of_6‚0.07894736528396606 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.4047618806362152 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2627362310886383 9"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4615384638309479 particle/accuracy_5_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.20649318397045135 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15176624059677124 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_3_of_3‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18213143944740295 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 9"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.2666666805744171 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5024321675300598 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1957012414932251 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.432753324508667 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09632395207881927 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4548107981681824 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3757576048374176 >&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2769171893596649 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19918137788772583 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13472221791744232 *jet/accuracy_1_of_1‚0.3476530909538269 ,particle/sensitivity‚0.25892308354377747 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02121024578809738 jet/accuracy_5_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0022755861282348633 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9960858225822449 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09675091505050659 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.17311827838420868 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 *jet/accuracy_2_of_3‚0.2023051679134369 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0063463118858635426 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.37037035822868347 ,jet/accuracy_3_of_4‚0.050389036536216736 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15654025971889496 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.003199421800673008 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36425042152404785 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.21580417454242706 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3303944170475006 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07069078087806702 jet/accuracy_5_of_6‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6071428656578064 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9983022212982178 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015232974663376808 9"Purity/1FRt0SRqqt3FBt/event_purity‚0.1428571492433548 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.14771826565265656 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9992668628692627 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8113547563552856 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1808559149503708 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.26994025707244873 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2396402209997177 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9924992322921753 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06627661734819412 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1723674088716507 +jet/accuracy_2_of_2‚0.08481646329164505 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.012107331305742264 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1086350604891777 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9355555772781372 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _step‚146 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07094132155179977 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.24235086143016815 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.00956221204251051 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928787350654602 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.00572300935164094 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2888888716697693 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.780701756477356 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9218106865882874 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4541403651237488 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14705882966518402 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3888889253139496 particle/accuracy_4_of_6‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013095238246023655 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.28340426087379456 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.1515967696905136 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.18602018058300018 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.33133381605148315 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9997790455818176 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 4 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937744140625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5403911471366882 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3055555522441864 jet/accuracy_4_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9335318207740784 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24582327902317047 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.012962514534592628 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.541963517665863 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2637864947319031 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.16108277440071106 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1281450241804123 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4108038544654846 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09567900747060776 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9857549667358398 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9927419424057007 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37938594818115234 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3355664312839508 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.20649318397045135 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9937398433685303 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9687500596046448 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.16093075275421143 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3752592206001282 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 +particle/specificity‚0.9438073039054871 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27083513140678406 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6000000238418579 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13526570796966553 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0714285746216774 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00946201104670763 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03242316097021103 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.19623656570911407 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.43351852893829346 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905875325202942 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15176624059677124 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2766329050064087 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.198849156498909 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4912281036376953 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34074074029922485 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4821525514125824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7631579041481018 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_6‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6236950755119324 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3205128312110901 particle/accuracy_6_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20862486958503723 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.18985970318317413 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9736841917037964 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5008876323699951 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3049033284187317 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1618036925792694 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4000000059604645 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9939814805984497 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2536400258541107 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9523809552192688 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3728441596031189 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 particle/accuracy_7_of_7‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.4047618806362152 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.314718633890152 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17920635640621185 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23696692287921906 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.26002180576324463 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5295237898826599 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_1_of_2‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1836468130350113 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.16060875356197357 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.17812588810920715 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9685760140419006 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.28630951046943665 *jet/accuracy_2_of_4‚0.3068970739841461 particle/accuracy_1_of_7‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45967742800712585 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.425595223903656 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6056048274040222 particle/accuracy_1_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20563317835330963 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936114549636841 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.027037037536501884 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9851189851760864 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 trainer/global_step‚1811 _runtime‚5069.855102707 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.24805986881256104 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21707820892333984 )jet/accuracy_1_of_3‚0.732528567314148 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9959677457809448 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.360965371131897 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 +"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.991028904914856 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.25550904870033264 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 jet/accuracy_4_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20775099098682404 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8040735125541687 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07183908671140671 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4883376359939575 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4341050982475281 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42895540595054626 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891875386238098 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.25806909799575806 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.26141852140426636 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0868760272860527 ,jet/accuracy_4_of_4‚0.004248607438057661 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14725035429000854 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4560374319553375 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9956240653991699 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9851189851760864 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 particle/accuracy_1_of_1‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2883598208427429 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3369311988353729 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.25833332538604736 particle/accuracy_1_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9603174924850464 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24042634665966034 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0803571417927742 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3637990355491638 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.393888920545578 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 % _timestamp‚1.7884862423110976e+09 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9827777743339539 jet/accuracy_4_of_5‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9985975027084351 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.006279607303440571 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016720136627554893 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6857299208641052 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3746213912963867 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 , Purity/1FRt3SRqqt*FBt/FBt_purity‚0.96875 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1854865998029709 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39578425884246826 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006989247165620327 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2338455319404602 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.47741931676864624 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.007635199464857578 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24052108824253082 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18411801755428314 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1991504728794098 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1386379450559616 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2031792849302292 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 epoch‚2 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5555555820465088 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09134220331907272 :"Purity/0FRt0SRqqt3FBt/event_purity‚0.10333333164453506 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2904040515422821 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.4385964870452881 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_2_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3888889253139496 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt*S³º©%Rqqt0FBt/SRqqt_purity‚0.20443622767925262 particle/accuracy_2_of_7‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01785714365541935 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9358238577842712 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2833111882209778 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.20378077030181885 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2083333283662796 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 jet/accuracy_2_of_7‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11491180956363678 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011111111380159855 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4615384638309479 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ."Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.15625 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939050078392029 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9995580911636353 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25259292125701904 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21229231357574463 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2554147243499756 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 )loss/val_total_loss‚47.15069997874852 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003150201519019902 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5035713911056519 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 *jet/accuracy_1_of_5‚0.8719742894172668 (particle/accuracy‚0.8368247747421265 6validation_average_jet_accuracy‚0.3366852402687073 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2178502231836319 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.27562788128852844 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24035803973674774 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5476190447807312 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2411607950925827 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.15457144379615784 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.4523809254169464 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003310351399704814 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5192307829856873 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25161290168762207 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9936347007751465 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.173611119389534 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.20588654279708862 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3729749619960785 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4064021110534668 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.10000000149011612 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0714285746216774 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5066303014755249 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 ,"Purity/1FRt*SRqqt3FBt/event_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3763798475265503 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.24659846723079681 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.386962354183197 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38967207074165344 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1496141254901886 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 particle/accuracy_2_of_4‚1 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚ 0.1640625 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚06‰v<ÍeÙ¦¥ :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07183908671140671 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4883376359939575 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4341050982475281 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42895540595054626 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891875386238098 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.25806909799575806 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.26141852140426636 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0868760272860527 ,jet/accuracy_4_of_4‚0.004248607438057661 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14725035429000854 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4560374319553375 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9956240653991699 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9851189851760864 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 particle/accuracy_1_of_1‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2883598208427429 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3369311988353729 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.25833332538604736 particle/accuracy_1_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9603174924850464 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24042634665966034 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0803571417927742 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3637990355491638 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.393888920545578 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 % _timestamp‚1.7884862423110976e+09 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9827777743339539 jet/accuracy_4_of_5‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9985975027084351 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.006279607303440571 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016720136627554893 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6857299208641052 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3746213912963867 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 , Purity/1FRt3SRqqt*FBt/FBt_purity‚0.96875 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1854865998029709 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39578425884246826 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006989247165620327 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2338455319404602 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.47741931676864624 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.007635199464857578 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24052108824253082 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18411801755428314 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1991504728794098 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1386379450559616 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2031792849302292 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 epoch‚2 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5555555820465088 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09134220331907272 :"Purity/0FRt0SRqqt3FBt/event_purity‚0.10333333164453506 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2904040515422821 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.4385964870452881 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_2_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3888889253139496 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.20443622767925262 particle/accuracy_2_of_7‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01785714365541935 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9358238577842712 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2833111882209778 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.20378077030181885 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2083333283662796 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 jet/accuracy_2_of_7‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11491180956363678 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011111111380159855 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4615384638309479 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ."Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.15625 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939050078392029 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9995580911636353 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25259292125701904 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21229231357574463 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2554147243499756 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 )loss/val_total_loss‚47.15069997874852 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003150201519019902 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5035713911056519 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 *jet/accuracy_1_of_5‚0.8719742894172668 (particle/accuracy‚0.8368247747421265 6validation_average_jet_accuracy‚0.3366852402687073 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2178502231836319 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.27562788128852844 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24035803973674774 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5476190447807312 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2411607950925827 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.15457144379615784 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.4523809254169464 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003310351399704814 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5192307829856873 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25161290168762207 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9936347007751465 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.173611119389534 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.20588654279708862 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3729749619960785 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4064021110534668 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.10000000149011612 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0714285746216774 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5066303014755249 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 ,"Purity/1FRt*SRqqt3FBt/event_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3763798475265503 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.24659846723079681 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.386962354183197 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38967207074165344 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1496141254901886 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 particle/accuracy_2_of_4‚1 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚ 0.1640625 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9900473952293396 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4100760221481323 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9957733154296875 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 *jet/accuracy_1_of_6‚0.9298246502876282 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.20690184831619263 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2044096440076828 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3888888657093048 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5833333134651184 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25298935174942017 6 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0483240932226181 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.01785714365541935 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.25420624017715454 *jet/accuracy_2_of_5‚0.3503551781177521 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9883745312690735 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45421159267425537 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558467626571655 >&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9772727489471436 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 particle/accuracy_2_of_3‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt2FBt/event_proportion‚0.008757560513913631 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.21131975948810577 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4451747536659241 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.2586805522441864 particle/accuracy_1_of_3‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.27454856038093567 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19466783106327057 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.28552383184432983 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23497183620929718 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 +jet/accuracy_3_of_6‚0.07894736528396606 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.4047618806362152 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2627362310886383 9"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4615384638309479 particle/accuracy_5_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.20649318397045135 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15176624059677124 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_3_of_3‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18213143944740295 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 9"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.2666666805744171 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5024321675300598 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1957012414932251 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.432753324508667 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09632395207881927 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4548107981681824 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3757576048374176 >&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2769171893596649 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19918137788772583 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13472221791744232 *jet/accuracy_1_of_1‚0.3476530909538269 ,particle/sensitivity‚0.25892308354377747 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02121024578809738 jet/accuracy_5_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0022755861282348633 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9960858225822449 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09675091505050659 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.17311827838420868 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 *jet/accuracy_2_of_3‚0.2023051679134369 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0063463118858635426 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.37037035822868347 ,jet/accuracy_3_of_4‚0.050389036536216736 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15654025971889496 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.003199421800673008 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36425042152404785 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.21580417454242706 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3303944170475006 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07069078087806702 jet/accuracy_5_of_6‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6071428656578064 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9983022212982178 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015232974663376808 9"Purity/1FRt0SRqqt3FBt/event_purity‚0.1428571492433548 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.14771826565265656 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9992668628692627 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8113547563552856 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1808559149503708 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.26994025707244873 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2396402209997177 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9924992322921753 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06627661734819412 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1723674088716507 +jet/accuracy_2_of_2‚0.08481646329164505 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.012107331305742264 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1086350604891777 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9355555772781372 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _step‚146 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07094132155179977 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.24235086143016815 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.00956221204251051 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928787350654602 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.00572300935164094 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2888888716697693 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.780701756477356 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9218106865882874 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4541403651237488 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14705882966518402 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3888889253139496 particle/accuracy_4_of_6‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013095238246023655 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.28340426087379456 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.1515967696905136 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.18602018058300018 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.33133381605148315 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9997790455818176 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 4 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937744140625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5403911471366882 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3055555522441864 jet/accuracy_4_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9335318207740784 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 8 Purity/*FRt1SRqqt*FéžpÎà,Bt/FRt_purity‚0.24582327902317047 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.012962514534592628 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.541963517665863 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2637864947319031 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.16108277440071106 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1281450241804123 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4108038544654846 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09567900747060776 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9857549667358398 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9927419424057007 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37938594818115234 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3355664312839508 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.20649318397045135 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9937398433685303 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9687500596046448 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.16093075275421143 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3752592206001282 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 +particle/specificity‚0.9438073039054871 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27083513140678406 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6000000238418579 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13526570796966553 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0714285746216774 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00946201104670763 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03242316097021103 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.19623656570911407 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.43351852893829346 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905875325202942 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15176624059677124 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2766329050064087 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.198849156498909 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4912281036376953 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34074074029922485 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4821525514125824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7631579041481018 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_6‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6236950755119324 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3205128312110901 particle/accuracy_6_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20862486958503723 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.18985970318317413 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9736841917037964 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5008876323699951 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3049033284187317 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1618036925792694 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4000000059604645 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9939814805984497 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2536400258541107 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9523809552192688 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3728441596031189 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 particle/accuracy_7_of_7‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.4047618806362152 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.314718633890152 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17920635640621185 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23696692287921906 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.26002180576324463 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5295237898826599 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_1_of_2‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1836468130350113 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.16060875356197357 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.17812588810920715 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9685760140419006 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.28630951046943665 *jet/accuracy_2_of_4‚0.3068970739841461 particle/accuracy_1_of_7‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45967742800712585 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.425595223903656 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6056048274040222 particle/accuracy_1_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20563317835330963 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936114549636841 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.027037037536501884 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9851189851760864 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 trainer/global_step‚1811 _runtime‚5069.855102707 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.24805986881256104 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21707820892333984 )jet/accuracy_1_of_3‚0.732528567314148 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9959677457809448 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.360965371131897 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 +"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.991028904914856 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.25550904870033264 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 jet/accuracy_4_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20775099098682404 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8040735125541687 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25’Ø&“œÚ‚(:‘ âÄèÔÓΡÝ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.507' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.701400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä]Ç>‚Û‚(:÷ âÄèÔ­‹ß& proc.memory.availableMB‚ 470.546875) proc.memory.percent‚20.74267864227295% disk./.usageGB‚1169.9367561340332$ disk.nvme0c0n1.in‚11077.46484375 disk.sda.in‚0 disk.sda.out‚ 153.3046875 disk.sda1.in‚0 disk.dm-1.out‚ 153.3046875$ disk.dm-0.out‚127556.37451171875" disk.nvme0n1.in‚11077.46484375 network.sent‚ 5.040037e+06( disk./.usagePercent‚78.5154373464774 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚97.12800979614258# proc.memory.rssMB‚ 3398.48046875 cpu‚25.330308761197603 proc.cpu.threads‚17) disk.nvme0c0n1.out‚127556.37451171875 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚11077.46484375 network.recv‚ 1.581139e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7265625 disk.sda2.out‚0' disk.nvme0n1.out‚127556.37451171875 disk.sda3.out‚ 153.3046875k »ñÜ‚(:… ñÄèÔË”äÞ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5008 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º,†ý€Ý‚(:õ ñÄèÔý¿ß disk.sda.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.588066e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 11211.78125 disk.sda3.out‚ 153.73046875 disk.dm-1.out‚ 153.73046875" proc.memory.rssMB‚ 3765.0859375 cpu‚25.329445099627847 network.sent‚ 5.145158e+06% disk./.usageGB‚1173.9447135925293 disk.drbd1260.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚135927.59130859375! disk.nvme0c0n1.in‚ 11211.78125) disk.nvme0c0n1.out‚135927.59130859375 disk.sda.out‚ 153.73046875) disk./.usagePercent‚78.78441473441737 disk.drbd1260.out‚ 0.7265625 disk.dm-0.in‚ 11211.78125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚71.13268375396729) proc.memory.availableMB‚ 4729.62109375* proc.memory.percent‚22.980260848999023$ disk.dm-0.out‚135927.59130859375 disk.sda3.in‚0,À}Þ‚(:ò €ÅèÔ‡©—à$ disk.dm-0.out‚136913.60302734375 disk.sda3.out‚ 154.2578125) disk.nvme0c0n1.out‚136913.60302734375# proc.memory.rssMB‚ 3397.49609375 disk.sda2.out‚0# memory_percent‚68.6267614364624) proc.memory.availableMB‚ 5140.19140625) proc.memory.percent‚20.73667049407959 cpu‚25.329542250032155 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.73046875 disk.sda2.in‚0 network.sent‚ 5.149244e+06 network.recv‚ 1.592801e+06) disk./.usagePercent‚78.78387686258793 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 154.2578125 disk.dm-0.in‚ 11211.78125 disk.nvme0n1.in‚ 11211.78125' disk.nvme0n1.out‚136913.60302734375 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 11211.78125% disk./.usageGB‚1173.9366989135742 disk.sda.out‚ 154.2578125šŽá“ß‚(:ˆ €ÅèÔýžáá gpu.0.gpu‚72.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚327.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ß•à‚(:ö ÅèÔ§¯ôá disk.sda3.in‚0 disk.sda3.out‚ 154.73046875# memory_percent‚99.9415397644043# proc.memory.rssMB‚ 3397.49609375% disk./.usageGB‚1173.9367218017578 disk.sda.in‚0 disk.sda.out‚ 154.73046875' disk.nvme0n1.out‚137257.38427734375$ proc.memory.availableMB‚9.578125 network.recv‚ 1.596251e+06 disk.dm-1.out‚ 154.73046875 disk.sda2.in‚0$ disk.dm-0.out‚137257.38427734375) proc.memory.percent‚20.73667049407959 cpu‚25.329391818275038 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.73046875 network.sent‚ 5.154344e+06# disk.nvme0c0n1.in‚ 11211.9453125) disk.nvme0c0n1.out‚137257.38427734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk./.usagePercent‚78.78387839863315 disk.drbd1260.in‚0 disk.dm-0.in‚ 11211.9453125! disk.nvme0n1.in‚ 11211.94531254zØÏá‚(:„ ÅèÔ†ºãá gpu.0.gpu‚97.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚49.0 gpu.0.powerWatts‚326.2' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.24 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…áÊ*vâ‚(:ë žÅèÔÇ•„á& proc.memory.availableMB‚ 9.56640625# proc.memory.rssMB‚ 3397.49609375 proc.cpu.threads‚17) disk./.usagePercent‚78.78387711859547 disk.dm-1.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚137696.49609375 disk.sda3.in‚0 disk.sda1.in‚0% disk./.usageGB‚1173.9367027282715 disk.drbd1260.in‚0 disk.dm-0.in‚ 11211.9453125 disk.sda3.out‚ 155.1328125# disk.nvme0c0n1.in‚ 11211.9453125 cpu‚25.329315685689455 network.sent‚ 5.161434e+06 network.recv‚ 1.599697e+06 disk.sda2.out‚0! disk.nvme0n1.in‚ 11211.9453125 disk.sda.out‚ 155.1328125 disk.sda1.out‚0$ memory_percent‚99.94161128997803) proc.memory.percent‚20.73667049407959 disk.dm-1.out‚ 155.1328125 disk.drbd1260.out‚0.734375! disk.dm-0.out‚137696.49609375& disk.nvme0c0n1.out‚137696.49609375 disk.sda.in‚0ÿ ñù“ã‚(:ˆ žÅèÔ½”ôá gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚240.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.1452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ªÞ,sä‚(:è ­ÅèÔÈîóá disk.sda2.out‚0$ memory_percent‚99.94158744812012) proc.memory.percent‚20.73667049407959 network.sent‚ 5.165714e+06) disk./.usagePercent‚78.78387839863315% disk./.usageGB‚1173.9367218017578 disk.dm-0.in‚ 11296.5078125 disk.dm-1.in‚0 disk.dm-1.out‚ 155.9140625 disk.drbd1260.out‚0.734375% proc.memory.availableMB‚ 9.5703125$ disk.nvme0n1.out‚137845.95703125# disk.nvme0c0n1.in‚ 11296.5078125 cpu‚25.329222817091107 proc.cpu.threads‚17 network.recv‚ 1.6045e+06! disk.dm-0.out‚137845.95703125 disk.sda.in‚0 disk.sda.out‚ 155.9140625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3397.49609375! disk.nvme0n1.in‚ 11296.5078125 disk.sda3.in‚0 disk.sda3.out‚ 155.9140625& disk.nvme0c0n1.out‚137845.95703125 disk.sda1.in‚0žÐ¡wžå‚(:“ ­ÅèÔƒ¨´â gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.024' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.604800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WPYzæ‚(:ï ¼ÅèÔ¯›ñá cpu‚25.32928242163675 disk.drbd1260.out‚ 0.73828125# disk.nvme0c0n1.in‚ 11370.0859375$ memory_percent‚99.94161128997803) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 11370.0859375 disk.sda3.out‚ 156.3359375 disk.sda.out‚ 156.3359375& proc.memory.availableMB‚ 9.56640625# proc.memory.rssMB‚ 3397.49609375 network.sent‚ 5.171606e+06 network.recv‚ 1.608695e+06& disk./.usagePercent‚78.78308861539% disk./.usageGB‚1173.9249534606934 disk.sda1.out‚0 disk.dm-1.out‚ 156.3359375 disk.drbd1260.in‚0# disk.dm-0.out‚138015.7724609375 disk.sda2.out‚0! disk.nvme0n1.in‚ 11370.0859375& disk.nvme0n1.out‚138015.7724609375 disk.sda3.in‚0( disk.nvme0c0n1.out‚138015.7724609375 disk.sda.in‚0ü|@ç‚(:„ ¼ÅèÔ”ñ›â gpu.0.gpu‚55.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚341.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)³…Á2è- _wandbruntime‚5173  _runtime‚5173²Š~ÓÅé¿ & lr-AdamW/pg2‚0.0008879653441296468 trainer/global_step‚1849 $ _timestamp‚1.788486346416636e+09 _runtime‚ 5173.95800728 _step‚147 & lr-AdamW/pg1‚0.0008879653441296468=>­ÊêÄ $ _timestamp‚1.788486346416636e+09 _runtime‚ 5173.95800728 _step‚147 & lr-AdamW/pg1‚0.0008879653441296468 & lr-AdamW/pg2‚0.0008879653441296468 trainer/global_step‚1849“]Ä‹2ë- _wandbruntime‚5173  _runtime‚5173…"Ž€û'ìõO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5644405792768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884863464177818e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚601710391434752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5644405792768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 _step‚148 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117700608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚582205299932068 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 trainer/global_step‚1849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚576572199541936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚576572174113968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601710391434752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26252574 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114552917654016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38907915 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607351202969088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121774275881984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26253321 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5640836962304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607354797227520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5633125818100 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84788203 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45906378 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 Be‡.íúO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5644405792768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884863464177818e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚601710391434752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5644405792768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 _step‚148 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117700608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚582205299932068 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 trainer/global_step‚1849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚576572199541936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚576572174113968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601710391434752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26252574 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114552917654016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38907915 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607351202969088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121774275881984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26253321 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5640836962304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607354797227520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5633125818100 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84788203 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45906378 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26266312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚576873845111168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5647400392704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121839720169984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607672382843904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚602024957023232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15910999 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7225170147840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚602024982451200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10356046 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15910266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115590144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114614463529984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10356049 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582513553707398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5639683168262 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5643831548928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 5176.15616134 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45904586 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26267048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45930696 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45904586 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114614434432000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _step‚149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5636116330714 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84858861 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1849 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38928165 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚582509961441882 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 % _timestamp‚1.7884863486143525e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5643831548928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84832745 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84858861 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45930696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86492160 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38928165Œlä 'òƒN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86492160 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38928165 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26266312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚576873845111168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5647400392704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121839720169984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607672382843904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚602024957023232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15910999 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7225170147840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚602024982451200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10356046 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15910266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115590144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114614463529984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10356049 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582513553707398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5639683168262 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5643831548928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 5176.15616134 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45904586 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26267048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45930696 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45904586 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114614434432000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _step‚149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5636116330714 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84858861 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1849 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38928165 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚582509961441882 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 % _timestamp‚1.7884863486143525e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5643831548928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84832745 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84858861 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45930696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224•®Q¤é2ó- _wandbruntime‚5176  _runtime‚5176Ò0gUôÏ 0loss/FBt1/assignment_loss‚0.2813815417716396 1loss/SRqqt1/detection_loss‚2.6148414278030394 /loss/SRqqt4/detection_loss‚0.86824897378403 % _timestamp‚1.7884863486168556e+09 /loss/FRt2/detection_loss‚2.3104914870875124 0loss/SRqqt2/detection_loss‚2.495950159761641 /loss/FBt2/detection_loss‚0.9466867294688912 -loss/FBt3/detection_loss‚2.46411269903183 1loss/SRqqt2/assignment_loss‚1.628110741244422 0loss/SRqqt4/assignment_loss‚1.12443565774737 .loss/FBt4/detection_loss‚2.230836643775304 0loss/FRt1/assignment_loss‚4.6263420626802265 /loss/FRt3/detection_loss‚2.1985517278313638 1loss/SRqqt3/detection_loss‚2.1544696663107192 epoch‚3 0loss/FRt2/assignment_loss‚3.0817379876971245 2loss/SRqqt1/assignment_loss‚1.6452987753227353 /loss/FRt4/detection_loss‚1.8275012279677594 0loss/FBt4/assignment_loss‚0.1633820223311583 /loss/FRt1/detection_loss‚3.1063600351225653 .loss/FBt1/detection_loss‚2.679014437728458 &loss/total_loss‚45.731398589258575 trainer/global_step‚1849 1loss/FBt3/assignment_loss‚0.16908751728514096 /loss/FRt3/assignment_loss‚3.711067500710487 0loss/FRt4/assignment_loss‚1.6300625892785878 0loss/FBt2/assignment_loss‚0.2082069358903017 1loss/SRqqt3/assignment_loss‚1.565220041626266 _runtime‚5176.158021858 _step‚150£‰]õÔ 2loss/SRqqt1/assignment_loss‚1.6452987753227353 /loss/FRt4/detection_loss‚1.8275012279677594 0loss/FBt4/assignment_loss‚0.1633820223311583 /loss/FRt1/detection_loss‚3.1063600351225653 .loss/FBt1/detection_loss‚2.679014437728458 &loss/total_loss‚45.731398589258575 trainer/global_step‚1849 1loss/FBt3/assignment_loss‚0.16908751728514096 /loss/FRt3/assignment_loss‚3.711067500710487 0loss/FRt4/assignment_loss‚1.6300625892785878 0loss/FBt2/assignment_loss‚0.2082069358903017 1loss/SRqqt3/assignment_loss‚1.565220041626266 _runtime‚5176.158021858 _step‚150 0loss/FBt1/assignment_loss‚0.2813815417716396 1loss/SRqqt1/detection_loss‚2.6148414278030394 /loss/SRqqt4/detection_loss‚0.86824897378403 % _timestamp‚1.7884863486168556e+09 /loss/FRt2/detection_loss‚2.3104914870875124 0loss/SRqqt2/detection_loss‚2.495950159761641 /loss/FBt2/detection_loss‚0.9466867294688912 -loss/FBt3/detection_loss‚2.46411269903183 1loss/SRqqt2/assignment_loss‚1.628ÍlRNH110741244422 0loss/SRqqt4/assignment_loss‚1.12443565774737 .loss/FBt4/detection_loss‚2.230836643775304 0loss/FRt1/assignment_loss‚4.6263420626802265 /loss/FRt3/detection_loss‚2.1985517278313638 1loss/SRqqt3/detection_loss‚2.1544696663107192 epoch‚3 0loss/FRt2/assignment_loss‚3.0817379876971245–fýV:uö‚(:ê ÚÅèÔø¦çà proc.cpu.threads‚17) disk./.usagePercent‚78.78230216024483! disk.nvme0n1.in‚ 11370.0859375% disk.nvme0n1.out‚138856.646484375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.7421875 disk.sda2.out‚0# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 network.sent‚ 5.251965e+06 network.recv‚ 1.617579e+06% disk./.usageGB‚1173.9132347106934 disk.sda3.in‚0 disk.dm-1.out‚ 157.14453125 disk.sda2.in‚0 disk.dm-0.in‚ 11370.0859375 cpu‚25.329268600625024' disk.nvme0c0n1.out‚138856.646484375 disk.sda.in‚0 disk.sda.out‚ 157.14453125 disk.dm-1.in‚0 disk.drbd1260.in‚0" disk.dm-0.out‚138856.646484375# memory_percent‚99.9420166015625 proc.memory.availableMB‚9.5 disk.sda3.out‚ 157.14453125# disk.nvme0c0n1.in‚ 11370.0859375 disk.sda1.out‚0?Jº“÷‚(:ˆ ÚÅèÔà¸íë gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚247.121' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.4242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0))lO“ø‚(:ˆ éÅèÔó©“Ý gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚225.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.1532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¿h(sù‚(:è éÅèÔÈüÎß disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7421875 cpu‚25.329197079763993 network.sent‚ 5.328752e+06 network.recv‚ 1.623483e+06! disk.nvme0n1.in‚ 11370.0859375' disk.nvme0c0n1.out‚139295.521484375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 157.59375" disk.dm-0.out‚139295.521484375 disk.sda3.in‚0 disk.sda.out‚ 157.59375 disk.sda2.out‚0) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17) disk./.usagePercent‚78.78387071840712% disk.nvme0n1.out‚139295.521484375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 11370.0859375$ memory_percent‚99.94359016418457% proc.memory.availableMB‚ 9.2421875# proc.memory.rssMB‚ 3397.49609375% disk./.usageGB‚1173.9366073608398 disk.sda3.out‚ 157.59375# disk.nvme0c0n1.in‚ 11370.0859375múYk“ú‚(:ˆ øÅèÔ„Ïãá gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚168.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b]˽û‚(:ô øÅèÔ¹¤”ã disk.sda3.out‚ 158.20703125 disk.sda.in‚0 disk.sda.out‚ 158.20703125) proc.memory.percent‚20.73667049407959 disk.dm-1.in‚0 disk.nvme0n1.in‚ 11515.796875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 11515.796875$ memory_percent‚99.94206428527832% proc.memory.availableMB‚ 9.4921875 cpu‚25.32926815871467 proc.cpu.threads‚17' disk.nvme0n1.out‚139447.28759765625 network.sent‚ 5.334711e+06) disk./.usagePercent‚78.78387071840712 disk.dm-1.out‚ 158.20703125 disk.drbd1260.out‚ 0.74609375 disk.sda2.out‚0 disk.dm-0.in‚ 11515.796875 disk.sda1.in‚0 network.recv‚ 1.628209e+06% disk./.usageGB‚1173.9366073608398 disk.sda2.in‚0) disk.nvme0c0n1.out‚139447.28759765625 disk.sda1.out‚0# proc.memory.rssMB‚ 3397.49609375 disk.drbd1260.in‚0$ disk.dm-0.out‚139447.28759765625 -ötü‚(:é ‡ÆèÔ‡¹¦á network.recv‚ 1.631942e+06) disk.nvme0c0n1.out‚139814.09228515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚158.5625 disk.drbd1260.out‚ 0.74609375 disk.sda2.out‚0$ disk.dm-0.out‚139814.09228515625) disk./.usagePercent‚78.78229576005647% disk./.usageGB‚1173.9131393432617 disk.sda.in‚0 disk.sda3.out‚158.5625# proc.memory.rssMB‚ 3397.49609375 cpu‚25.32920421902852 proc.cpu.threads‚17 network.sent‚ 5.340143e+06" disk.nvme0c0n1.in‚ 11515.796875 disk.sda.out‚158.5625 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚99.94361400604248& proc.memory.availableMB‚ 9.23828125) proc.memory.percent‚20.73667049407959 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 11515.796875 disk.nvme0n1.in‚ 11515.796875' disk.nvme0n1.out‚139814.09228515625‹tjP“ý‚(:ˆ ‡ÆèÔÔÿâ gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚241.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01crÝþ‚(:’ –ÆèÔð¯«Ý gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.034' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.60679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îRuÿ‚(:ê –ÆèÔç…´Þ disk.drbd1260.out‚0.75 disk.sda2.out‚0$ memory_percent‚99.94208812713623) disk./.usagePercent‚78.74379888310263 disk.sda3.out‚ 159.00390625' disk.nvme0c0n1.out‚140156.927734375 disk.dm-0.in‚ 11515.796875& proc.memory.availableMB‚ 9.48828125# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 cpu‚25.329150047421063 network.recv‚ 1.635514e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 159.00390625" disk.dm-0.out‚140156.927734375 proc.cpu.threads‚17 network.sent‚ 5.345261e+06 disk.nvme0n1.in‚ 11515.796875% disk.nvme0n1.out‚140156.927734375" disk.nvme0c0n1.in‚ 11515.796875 disk.sda.out‚ 159.00390625 disk.sda2.in‚0% disk./.usageGB‚1173.3395080566406 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0V»í%“€‚(:ˆ ¥ÆèÔŠ¡ÔÝ gpu.0.gpu‚47.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.857' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬V&íh‚(:Ý ¥ÆèÔ–àÏß disk.sda.out‚159.3125% disk./.usageGB‚1173.3391304016113 disk.sda3.out‚159.3125 disk.sda1.in‚0# proc.memory.rssMB‚ 3397.49609375 proc.cpu.threads‚17 network.sent‚ 5.351405e+06 disk.dm-0.in‚ 11515.796875 disk.nvme0n1.in‚ 11515.796875' disk.nvme0c0n1.out‚140681.900390625 disk.dm-1.in‚0 disk.dm-1.out‚159.3125 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.74377353835673" disk.dm-0.out‚140681.900390625% disk.nvme0n1.out‚140681.900390625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 11515.796875 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.75 network.recv‚ 1.640421e+06 disk.sda1.out‚0$ memory_percent‚99.94513988494873& proc.memory.availableMB‚ 8.98828125) proc.memory.percent‚20.73667049407959 cpu‚25.32914981911684ïC”€’‚‚(:‡ ´ÆèÔÇœ›Ý gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚241.545' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.309 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅAwƒ‚(:ì ´ÆèÔ®Õþß( disk./.usagePercent‚78.7437763544396 disk.dm-0.in‚ 11678.0625# disk.dm-0.out‚141066.8271484375& disk.nvme0n1.out‚141066.8271484375( disk.nvme0c0n1.out‚141066.8271484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 11678.0625 disk.sda3.out‚ 159.82421875 disk.sda.in‚0 disk.dm-1.in‚0$ memory_percent‚99.94509220123291& proc.memory.availableMB‚ 8.99609375# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 network.recv‚ 1.644761e+06% disk./.usageGB‚1173.3391723632812 disk.sda.out‚ 159.82421875 disk.sda2.in‚0 network.sent‚ 5.357407e+06 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 11678.0625 disk.dm-1.out‚ 159.82421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.75390625 cpu‚25.329157925536546 proc.cpu.threads‚17Ïs0ž2„- _wandbruntime‚5293  _runtime‚5293hþªÇ…Á & lr-AdamW/pg1‚0.0008781566181476488 & lr-AdamW/pg2‚0.0008781566181476488 trainer/global_step‚1899 % _timestamp‚1.7884864655897193e+09 _runtime‚5293.131563486 _step‚151á¢&wÌ†Æ & lr-AdamW/pg1‚0.0008781566181476488 & lr-AdamW/pg2‚0.0008781566181476488 trainer/global_step‚1899 % _timestamp‚1.7884864655897193e+09 _runtime‚5293.131563486 _step‚151—»©Ô2‡- _wandbruntime‚5293  _runtime‚5293ª±½bû'ˆõO KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚623230510950912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47098246 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117630182354432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591655749401936 _step‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617439942257152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚623234105209344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5790594121728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26941491 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26940702 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597441994860084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10624765 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚623234105209344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 _runtime‚5293.134334952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87044745 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10624762 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117630211452416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7414359268864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617439916829184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39920415 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591655749401936 _step‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617439942257152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚623234105209344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5790594121728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26941491 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26940702 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597441994860084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10624765 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚623234105209344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 _runtime‚5293.134334952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87044745 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10624762 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117630211452416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7414359268864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617439916829184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39920415 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚623230510950912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931˜ú“Š‚(:ˆ ÃÆèÔ”çûÝ gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚237.763' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÿµœp‹‚(:å ÃÆèÔûšà) disk./.usagePercent‚78.74377507440194$ disk./.usageGB‚1173.339153289795 disk.nvme0c0n1.in‚ 11678.0625# disk.dm-0.out‚141186.1943359375 disk.nvme0n1.in‚ 11678.0625 disk.sda3.out‚ 160.234375) proc.memory.percent‚20.73667049407959 network.sent‚ 5.375733e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 160.234375 disk.sda2.out‚0 cpu‚25.32907448218225 proc.cpu.threads‚17 network.recv‚ 1.648551e+06 disk.sda.in‚0 disk.sda.out‚ 160.234375 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚141186.1943359375 disk.sda3.in‚0$ memory_percent‚99.94661808013916( disk.nvme0c0n1.out‚141186.1943359375 disk.drbd1260.out‚ 0.75390625 disk.sda2.in‚0 disk.dm-0.in‚ 11678.0625& proc.memory.availableMB‚ 8.74609375# proc.memory.rssMB‚ 3397.49609375 disk.drbd1260.in‚01&—R2Œ- _wandbruntime‚5295  _runtime‚5295.mÄ¥þM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617754507845632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5793588708352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126075904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚591957420399136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954345 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ]KDeviceStatsMonitor.on_train_batch_end/inRŽ_#active_split_bytes.small_pool.freed‚ 7418096507904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1899 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚623548096553984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617754533273600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597743064104590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10630246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591957394971168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5785669133422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 $ _timestamp‚1.788486467917162e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚623551690825728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125109824738304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5789235970970 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚153 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16324099 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39940659 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47148648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87089313 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10630249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16324896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96977920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5295.460511188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87089313 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39940665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47148648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26955145 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39940665 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5793588708352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597746656370106 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47122538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚797Ä Ì= 'ŽƒN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5295.460511188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87089313 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39940665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47148648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26955145 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39940665 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5793588708352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597746656370106 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47122538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚797 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617754507845632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5793588708352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126075904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚591957420399136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954345 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7418096507904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1899 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚623548096553984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617754533273600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597743064104590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10630246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591957394971168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5785669133422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 $ _timestamp‚1.788486467917162e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚623551690825728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125109824738304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5789235970970 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚153 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16324099 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39940659 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47148648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87089313 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10630249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16324896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96977920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0™EQ2- _wandbruntime‚5295  _runtime‚5295®yüWÑ 0loss/FRt4/assignment_loss‚1.7350797278936518 0loss/FBt1/assignment_loss‚0.6154690098611911 /loss/FRt2/detection_loss‚2.3264026642264275 /loss/FBt1/detection_loss‚2.4698750145973696 .loss/FBt3/detection_loss‚2.276326556416119 /loss/FBt4/detection_loss‚2.2047722807952335 %loss/total_loss‚44.92888355121515 1loss/SRqqt2/detection_loss‚2.6517024669382305 1loss/SRqqt3/detection_loss‚2.0656100664287806 1loss/SRqqt4/detection_loss‚0.9724701941599996 /loss/FRt3/assignment_loss‚3.591101979925519 1loss/SRqqt2/assignment_loss‚2.013581123099559 1loss/FBt2/assignment_loss‚0.25776774562988075 .loss/FRt1/detection_loss‚3.027296710897375 /loss/FRt3/detection_loss‚2.1472479415436587 0loss/FRt2/assignment_loss‚2.4906163858563715 1loss/SRqqt1/assignment_loss‚1.286566286526145 1loss/SRqqt4/assignment_loss‚1.006038977812213 .loss/FRt4/detection_loss‚1.808517370286376 0loss/FRt1/assignment_loss‚3.9954182792592934 _step‚154 trainer/global_step‚1899 0loss/FBt4/assignment_loss‚0.1723744003916181 1loss/FBt3/assignment_loss‚0.40033714001508675 1loss/SRqqt1/detection_loss‚2.9803080343185586 _runtime‚ 5295.4626227 1loss/SRqqt3/assignment_loss‚1.540181792061776 /loss/FBt2/detection_loss‚0.8938214022747201 epoch‚3 % _timestamp‚1.7884864679210725e+09¥,‘\‘Ö 1loss/SRqqt4/assignment_loss‚1.006038977812213 .loss/FRt4/detection_loss‚1.808517370286376 0loss/FRt1/assignment_loss‚3.9954182792592934 _step‚154 trainer/global_step‚1899 0loss/FBt4/assignment_loss‚0.1723744003916181 1loss/FBt3/assignment_loss‚0.40033714001508675 1loss/SRqqt1/detection_loss‚2.9803080343185586 _runtime‚ 5295.4626227 1loss/SRqqt3/assignment_loss‚1.540181792061776 /loss/FBt2/detection_loss‚0.8938214022747201 epoch‚3 % _timestamp‚1.7884864679210725e+09 0loss/FRt4/assignment_loss‚1.7350797278936518 0loss/FBt1/assignment_loss‚0.6154690098611911 /loss/FRt2/detection_loss‚2.3264026642264275 /loss/FBt1/detection_loss‚2.4698750145973696 .loss/FBt3/detection_loss‚2.276326556416119 /loss/FBt4/detection_loss‚2.2047722807952335 %loss/total_loss‚44.92888355121515 1loss/SRqqt2/detection_loss‚2.6517024669382305 1loss/SRqqt3/detection_loss‚2.0656100664287806 1loss/SRqqt4/detection_loss‚0.9724701941599996 /loss/FRt3/assignment_loss‚3.591101979925519 1loss/SRqqt2/assignment_loss‚2.013581123099559 1loss/FBt2/assignment_loss‚0.25776774562988075 .loss/FRt1/detection_loss‚3.027296710897375 /loss/FRt3/detection_loss‚2.1472479415436587 0loss/FRt2/assignment_loss‚2.4906163858563715 1loss/SRqqt1/assignment_loss‚1.286566286526145š;Æxit’‚(:é ÒÆèÔÄô¦ß network.sent‚ 5.436726e+06) disk./.usagePercent‚78.74377661044714 disk.dm-0.in‚ 11678.0625 disk.nvme0n1.in‚ 11678.0625 disk.sda3.in‚0 disk.sda3.out‚ 160.60546875$ memory_percent‚99.94382858276367$ proc.memory.availableMB‚9.203125% disk./.usageGB‚1173.3391761779785 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.32904880304429( disk.nvme0c0n1.out‚141379.1572265625 disk.sda.in‚0 disk.sda.out‚ 160.60546875 disk.drbd1260.in‚0& disk.nvme0n1.out‚141379.1572265625# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 network.recv‚ 1.653302e+06 disk.nvme0c0n1.in‚ 11678.0625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 160.60546875 disk.drbd1260.out‚ 0.7578125 disk.sda2.in‚0# disk.dm-0.out‚141379.1572265625Å]7““‚(:ˆ ÒÆèÔî—óß gpu.0.gpu‚99.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚338.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Û6v”‚(:ë áÆèÔð¼Úá disk.sda.out‚ 161.0703125$ memory_percent‚99.94232654571533& proc.memory.availableMB‚ 9.44921875 disk.dm-0.in‚ 11678.0625$ disk.dm-0.out‚141836.49755859375 disk.dm-1.out‚ 161.0703125 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 11678.0625 disk.nvme0c0n1.in‚ 11678.0625 disk.sda.in‚0) proc.memory.percent‚20.73667049407959 disk.sda3.out‚ 161.0703125# proc.memory.rssMB‚ 3397.49609375 cpu‚25.329129835197282 proc.cpu.threads‚17 disk.sda2.in‚0 network.recv‚ 1.659532e+06) disk./.usagePercent‚78.74256902290813 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7578125' disk.nvme0n1.out‚141837.87255859375 network.sent‚ 5.51551e+06% disk./.usageGB‚1173.3211822509766 disk.sda2.out‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚141837.49755859375½D$6•‚(:’ áÆèÔ©Åñå gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.092' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.81840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w¶,"–‚(:’ ðÆèÔíǨÞ gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.11' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.422000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K¼º€—‚(:õ ðÆèÔè÷êß disk.sda.in‚0 disk.sda.out‚ 161.6484375 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 161.6484375 disk.drbd1260.out‚ 0.76171875 disk.sda2.out‚0$ memory_percent‚99.94235038757324# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 cpu‚25.329021227872097 network.sent‚ 5.519453e+06) disk./.usagePercent‚78.73774430491784% disk./.usageGB‚1173.2492904663086( disk.nvme0c0n1.out‚142036.5400390625 disk.dm-1.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 9.4453125 proc.cpu.threads‚17 disk.dm-0.in‚11819.84765625# disk.dm-0.out‚142036.5400390625 disk.sda3.out‚ 161.6484375 disk.sda1.in‚0 network.recv‚ 1.664064e+06" disk.nvme0n1.in‚11819.84765625& disk.nvme0n1.out‚142036.5400390625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚11819.84765625”\’˜‚(:‡ ÿÆèÔäðÆÜ gpu.0.gpu‚26.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚333.311' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6622 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Át™‚(:ö ÿÆèÔéèÊÞ$ memory_percent‚99.94537830352783& proc.memory.availableMB‚ 8.94921875 cpu‚25.329045387368115 network.recv‚ 1.667524e+06% disk./.usageGB‚1173.2493438720703 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.76171875 disk.sda2.in‚0# disk.dm-0.out‚142421.3525390625 disk.sda3.out‚ 162.0703125$ disk.nvme0c0n1.in‚11819.84765625 disk.dm-1.out‚ 162.0703125 disk.dm-0.in‚11819.84765625" disk.nvme0n1.in‚11819.84765625& disk.nvme0n1.out‚142421.3525390625 disk.sda3.in‚0( disk.nvme0c0n1.out‚142421.3525390625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 network.sent‚ 5.524555e+06) disk./.usagePercent‚78.73774788902331 disk.dm-1.in‚0 disk.sda.out‚ 162.0703125 disk.sda1.in‚07Ë“š‚(:ˆ ŽÇèÔà•ãá gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚243.624' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.7248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µþì‚›‚(:÷ ŽÇèÔ…Í„å proc.cpu.threads‚17% disk./.usageGB‚1171.9105033874512 disk.drbd1260.in‚0# disk.dm-0.out‚142727.2646484375& disk.nvme0n1.out‚142727.2646484375 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚99.94537830352783) disk./.usagePercent‚78.64789718073854 disk.dm-1.in‚0 disk.dm-1.out‚ 162.41796875 disk.dm-0.in‚11819.84765625" disk.nvme0n1.in‚11819.84765625$ disk.nvme0c0n1.in‚11819.84765625( disk.nvme0c0n1.out‚142727.2646484375 disk.sda1.out‚0 network.sent‚ 5.529578e+06 network.recv‚ 1.671126e+06 disk.drbd1260.out‚0.765625 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 162.41796875& proc.memory.availableMB‚ 8.94921875# proc.memory.rssMB‚ 3397.49609375 disk.sda2.in‚0 disk.sda.out‚ 162.41796875) proc.memory.percent‚20.73667049407959 cpu‚25.328948402392314äHÑ‚œ‚(:÷ ÇèÔ¥¡•á disk.sda3.in‚0 disk.sda3.out‚ 162.83984375$ memory_percent‚99.94690418243408& proc.memory.availableMB‚ 8.69921875) disk./.usagePercent‚78.57838857511695% disk./.usageGB‚1170.8747749328613 disk.dm-0.in‚11819.84765625# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 network.recv‚ 1.675899e+06$ disk.nvme0c0n1.in‚11819.84765625( disk.nvme0c0n1.out‚142919.9443359375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚11819.84765625 disk.sda.out‚ 162.83984375 disk.dm-1.out‚ 162.83984375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.765625 disk.sda2.out‚0# disk.dm-0.out‚142919.9443359375& disk.nvme0n1.out‚142919.9443359375 cpu‚25.328998188052775 network.sent‚ 5.537857e+06 disk.sda1.in‚0÷KØÚ‚(:„ ÇèÔì¬îá gpu.0.gpu‚92.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚117.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.54 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç‹ Ø“ž‚(:ˆ ¬ÇèÔųßá gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚248.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D‡á%ƒŸ‚(:ø ¬ÇèÔ´°¡ã' disk.nvme0n1.out‚143281.16162109375# proc.memory.rssMB‚ 3397.49609375) disk./.usagePercent‚78.57706527217246 disk.sda3.in‚0 disk.sda3.out‚ 163.40234375) disk.nvme0c0n1.out‚143281.16162109375 disk.sda1.in‚0 disk.dm-0.in‚11926.62890625$ disk.dm-0.out‚143281.16162109375& proc.memory.availableMB‚ 9.19140625 network.recv‚ 1.680093e+06$ disk.nvme0c0n1.in‚11926.62890625 disk.sda.in‚0 disk.dm-1.out‚ 163.40234375 disk.drbd1260.out‚ 0.76953125# memory_percent‚99.9439001083374) proc.memory.percent‚20.73667049407959 network.sent‚ 5.5418e+06% disk./.usageGB‚1170.8550567626953 disk.sda.out‚ 163.40234375 disk.sda1.out‚0 disk.dm-1.in‚0 cpu‚25.32889865933045 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚11926.62890625ÇÂ:‰2 - _wandbruntime‚5410  _runtime‚5410ñpiïÇ¡Á trainer/global_step‚1949 % _timestamp‚1.7884865827191293e+09 _runtime‚5410.260529228 _step‚155 & lr-AdamW/pg1‚0.0008680378293184004 & lr-AdamW/pg2‚0.0008680378293184004ÆìÃÌ¢Æ & lr-AdamW/pg1‚0.0008680378293184004 & lr-AdamW/pg2‚0.0008680378293184004 trainer/global_step‚1949 % _timestamp‚1.7884865827191293e+09 _runtime‚5410.260529228 _step‚155›³µ£Ã2£- _wandbruntime‚5410  _runtime‚5410lTˆQŽ¤õO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89249107 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5940351281152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚633169493079552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 [KDeviceStatsMonitor.on_train_batch_startBÁ3Âm#/reserved_bytes.large_pool.allocated‚ 53397684224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚633169493079552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚633169467651584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16728858 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89275197 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27628587 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚767 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40932909 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40932909 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48342282 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48316198 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90699776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48342282 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639113413191168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40932915 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128313925065728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639109818932736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5940351281152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128314044863488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606448912896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639113413191168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5943920111616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 trainer/global_step‚1949 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚633169467651584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10898965 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40932915 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 % _timestamp‚1.7884865827203274e+09 _runtime‚5410.262018066 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚606739273833968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639109818932736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27627820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120707476152832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 _step‚156 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0O\¤®(¥úO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48342282 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639113413191168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40932915 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128313925065728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639109818932736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5940351281152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128314044863488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606448912896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639113413191168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5943920111616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 trainer/global_step‚1949 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚633169467651584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10898965 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40932915 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 % _timestamp‚1.7884865827203274e+09 _runtime‚5410.262018066 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚606739273833968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639109818932736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27627820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120707476152832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 _step‚156 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89249107 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5940351281152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚633169493079552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚633169493079552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚633169467651584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16728858 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89275197 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27628587 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚767 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40932909 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40932909 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48342282 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48316198 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90699776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224œ;È82¦- _wandbruntime‚5412  _runtime‚5412_¿Gš'§ÿM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚633484084096000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40953165 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚764 _runtime‚5412.462473625 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607040970259136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89319765 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27641517 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16737835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89293649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚633484058668032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639430998807552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128379330145792 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚633484084096000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612976166767298 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48366600 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7610398803456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5938788773678 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89293649 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27642284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607040944831168 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639430998807552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚633484058668032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639427404535808 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119784448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10904449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120769051126784 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120769022028800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612979759032814 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48366600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48340490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 % _timestamp‚1.7884865849205542e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40953165 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89293649 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27642284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607040944831168 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639430998807552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚633484058668032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639427404535808 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119784448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10904449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120769051126784 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120769022028800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612979759032814 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 I@DeviceéL˜ŠÑStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48366600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48340490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 % _timestamp‚1.7884865849205542e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40953165 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89319765 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27641517 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16737835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89293649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚633484058668032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639430998807552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128379330145792 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚633484084096000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612976166767298 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48366600 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7610398803456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5938788773678 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123784799049216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7799381715968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621822849121936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17142418 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49534150 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 trainer/global_step‚1999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41945409 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚654992721172992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚648899018473984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128186368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91479559 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91505649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7799282627584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654992721172992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚648899018473984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6085351063564 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654989126914560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚648899043901952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49534150 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49560234 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28315580 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672[à}oÛÁúO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123784799049216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7799381715968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621822849121936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17142418 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49534150 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 trainer/global_step‚1999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41945409 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚654992721172992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚648899018473984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128186368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91479559 M@DeviceStatsMonitor.on_train_bÐj¥% atch_start/allocation.all.allocated‚91505649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7799282627584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654992721172992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚648899018473984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6085351063564 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654989126914560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚648899043901952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49534150 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49560234 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28315580 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41945409 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚818 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 _step‚160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚648899043901952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49560234 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17150689 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7803119874560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6088341576386 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 trainer/global_step‚1999 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655306712517632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91524101 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41965665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128173056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41965665 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11178649 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655310306789376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649213634918400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649213609490432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91524101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚628209269430006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11178646 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49558442 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28330152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5530.577969941 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6096671870976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123846315827200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚622124520119136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6093103027200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131649435701760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6084774738838 % _timestamp‚1.7884867030359223e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91550217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49584552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649213634918400 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655310306789376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6096671870976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7803218949632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚628212861695522 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6093103027200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41965659 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649213609490432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49584552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28329335 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚814 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91550217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131649563874816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99075072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11178649 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655310306789376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649213634918400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649213609490432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91524101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚628209269430006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11178646 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49558442 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28330152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5530.577969941 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6096671870976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123846315827200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚622124520119136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6093103027200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131649435701760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6084774738838 % _timestamp‚1.7884867030359223e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91550217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49584552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649213634918400 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655310306789376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6096671870976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7803218949632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚628212861695522 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6093103027200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41965659 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649213609490432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49584552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28329335 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚814 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91550217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131649563874816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99075072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¡ßóës2Å- _wandbruntime‚5530  _runtime‚5530ŲMÚXÆÒ 1loss/SRqqt4/assignment_loss‚1.126581447166721 /loss/FRt3/detection_loss‚2.2946235393903343 0loss/SRqqt2/detection_loss‚2.450283098479976 /loss/FRt1/assignment_loss‚4.687886782720978 /loss/FBt4/detection_loss‚2.7468191411184226 epoch‚3 0loss/FRt4/assignment_loss‚1.8547176296980876 1loss/SRqqt2/assignment_loss‚1.874505549021389 2loss/SRqqt3/assignment_loss‚1.5825904069782852 .loss/FRt1/detection_loss‚3.091494804503871 0loss/SRqqt1/detection_loss‚2.637121303975582 /loss/FRt2/assignment_loss‚2.553838265496631 1loss/SRqqt3/detection_loss‚1.9814485268933433 /loss/FBt3/detection_loss‚2.5291162651998027 &loss/total_loss‚46.996751941532494 _runtime‚5530.579723197 trainer/global_step‚1999 0loss/FRt3/assignment_loss‚3.9118600193276465 0loss/FBt1/assignment_loss‚0.3418047212418612 0loss/FBt3/assignment_loss‚0.2784735616226465 1loss/SRqqt4/detection_loss‚0.9825194731731958 /loss/FBt2/detection_loss‚0.9978998537858828 0loss/FBt4/assignment_loss‚0.2514459221198669 /loss/FRt4/detection_loss‚1.8391541519360441 .loss/FBt1/detection_loss‚2.686879668916975 2loss/SRqqt1/assignment_loss‚1.9630921203643084 1loss/FBt2/assignment_loss‚0.18381007665610402 .loss/FRt2/detection_loss‚2.148785611744537 % _timestamp‚1.7884867030385635e+09 _step‚1623ƒZ¶]Ç× &loss/total_loss‚46.996751941532494 _runtime‚5530.579723197 trainer/global_step‚1999 0loss/FRt3/assignment_loss‚3.9118600193276465 0loss/FBt1/assignment_loss‚0.3418047212418612 0loss/FBt3/assignment_loss‚0.2784735616226465 1loss/SRqqt4/detection_loss‚0.9825194731731958 /loss/FBt2/detection_loss‚0.9978998537858828 0loss/FBt4/assignment_loss‚0.2514459221198669 /loss/FRt4/detection_loss‚1.8391541519360441 .loss/FBt1/detection_loss‚2.686879668916975 2loss/SRqqt1/assignment_loss‚1.9630921203643084 1loss/FBt2/assignment_loss‚0.18381007665610402 .loss/FRt2/detection_loss‚2.148785611744537 % _timestamp‚1.7884867030385635e+09 _step‚162 1loss/SRqqt4/assignment_loss‚1.126581447166721 /loss/FRt3/detection_loss‚2.2946235393903343 0loss/SRqqt2/detection_loss‚2.450283098479976 /loss/FRt1/assignment_loss‚4.687886782720978 /loss/FBt4/detection_loss‚2.7468191411184226 epoch‚3 0loss/FRt4/assignment_loss‚1.8547176296980876 1loss/SRqqt2/assignment_loss‚1.874505549021389 2loss/SRqqt3/assignment_loss‚1.5825904069782852 .loss/FRt1/detection_loss‚3.091494804503871 0loss/SRqqt1/detection_loss‚2.637121303975582 /loss/FRt2/assignment_loss‚2.553838265496631 1loss/SRqqt3/detection_loss‚1.9814485268933433 /loss/FBt3/detection_loss‚2.5291162651998027¢†V{´yÈ‚(:î ³ÈèÔ—À¡ß' disk.nvme0n1.out‚145820.61083984375% proc.memory.availableMB‚ 9.4453125) proc.memory.percent‚20.73667049407959 network.recv‚ 1.718986e+06$ disk./.usageGB‚1171.335033416748 disk.sda3.in‚0 disk.sda3.out‚ 167.34375# disk.nvme0c0n1.in‚ 12243.0546875$ disk.dm-0.out‚145820.61083984375! disk.nvme0n1.in‚ 12243.0546875) disk./.usagePercent‚78.60927690815318 disk.drbd1260.out‚ 0.78515625 disk.sda2.in‚0 disk.dm-0.in‚ 12243.0546875$ memory_percent‚99.94235038757324# proc.memory.rssMB‚ 3397.49609375 cpu‚25.328507962068883 proc.cpu.threads‚17 network.sent‚ 5.673155e+06) disk.nvme0c0n1.out‚145820.61083984375 disk.sda.in‚0 disk.sda.out‚ 167.34375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 167.34375 disk.drbd1260.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0Ì°kižÉ‚(:“ ³ÈèÔìÉãß gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.155' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.431000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤~ãVÊ‚(:í ÂÈèÔ¨ùÁß# proc.memory.rssMB‚ 3397.49609375# disk.dm-0.out‚146122.4931640625& disk.nvme0n1.out‚146122.4931640625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 12243.0546875 disk.sda2.in‚0% proc.memory.availableMB‚ 9.4453125 proc.cpu.threads‚17 network.sent‚ 5.759416e+06! disk.nvme0n1.in‚ 12243.054¸uþ"6875( disk.nvme0c0n1.out‚146122.4931640625 disk.sda.out‚ 167.765625 disk.sda1.in‚0 disk.dm-1.out‚ 167.765625 disk.sda2.out‚0$ memory_percent‚99.94235038757324 network.recv‚ 1.724793e+06) disk./.usagePercent‚78.61758614469203 disk.sda3.out‚ 167.765625 disk.sda.in‚0) proc.memory.percent‚20.73667049407959 cpu‚25.32851855817483% disk./.usageGB‚1171.4588470458984 disk.dm-0.in‚ 12243.0546875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7890625YçmY“Ë‚(:ˆ ÂÈèÔÞšöà gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚338.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÆ·~Ì‚(:ó ÑÈèÔ½£‘à network.recv‚ 1.72973e+06% disk./.usageGB‚1171.4588661193848# disk.nvme0c0n1.in‚ 12243.0546875 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚146271.9658203125! disk.nvme0n1.in‚ 12243.0546875 disk.sda3.out‚ 168.10546875( disk.nvme0c0n1.out‚146271.9658203125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7890625 disk.dm-0.in‚ 12243.0546875% proc.memory.availableMB‚ 8.4140625 proc.cpu.threads‚17) disk./.usagePercent‚78.61758742472969 disk.sda.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚146271.9658203125) proc.memory.percent‚20.73667049407959 cpu‚25.328525529879084 network.sent‚ 5.765762e+06 disk.sda.out‚ 168.10546875 disk.dm-1.out‚ 168.10546875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚99.94864463806152# proc.memory.rssMB‚ 3397.49609375V¯ *“Í‚(:ˆ ÑÈèÔ¹ºØá gpu.0.gpu‚98.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚194.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.9852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TØ=ƒÎ‚(:ø àÈèÔ‚â¨ß disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.out‚ 0.79296875 disk.sda3.out‚ 168.66796875$ memory_percent‚99.94261264801025# proc.memory.rssMB‚ 3397.49609375 disk.dm-1.in‚0 disk.dm-1.out‚ 168.66796875& proc.memory.availableMB‚ 9.40234375 cpu‚25.328545917836102 proc.cpu.threads‚17 network.recv‚ 1.734325e+06 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 12243.0546875 disk.sda3.in‚0) proc.memory.percent‚20.73667049407959 network.sent‚ 5.771616e+06) disk./.usagePercent‚78.61679636144888# disk.nvme0c0n1.in‚ 12243.0546875 disk.sda.out‚ 168.66796875 disk.sda2.out‚0$ disk.dm-0.out‚146518.83447265625! disk.nvme0n1.in‚ 12243.0546875' disk.nvme0n1.out‚146518.83447265625$ disk./.usageGB‚1171.447078704834) disk.nvme0c0n1.out‚146518.834472656255ÁCï“Ï‚(:ˆ àÈèÔܶûá gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚230.054' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.0108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«±¥cyЂ(:î ïÈèÔ …ëß disk.sda.in‚0 disk.dm-1.out‚ 169.0859375 disk.dm-0.in‚ 12456.359375# proc.memory.rssMB‚ 3397.49609375 cpu‚25.328421433208746$ disk./.usageGB‚1171.960838317871 disk.sda3.out‚ 169.0859375) disk.nvme0c0n1.out‚146734.29150390625 disk.sda2.out‚0# proc.memory.availableMB‚8.90625) proc.memory.percent‚20.73667049407959 network.sent‚ 5.776718e+06) disk./.usagePercent‚78.65127520015284 disk.sda.out‚ 169.0859375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.79296875$ memory_percent‚99.94564056396484 network.recv‚ 1.73768e+06" disk.nvme0c0n1.in‚ 12456.359375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚146734.29150390625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 12456.359375' disk.nvme0n1.out‚146734.29150390625 disk.sda3.in‚0+Aôi“Ñ‚(:ˆ ïÈèÔÍ¡®à gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚226.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.3658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=­ø[}Ò‚(:ò þÈèÔ×üÿà disk.sda2.in‚0' disk.nvme0n1.out‚147060.27685546875 disk.sda3.out‚ 169.36328125 network.recv‚ 1.741307e+06) disk.nvme0c0n1.out‚147060.27685546875 disk.drbd1260.in‚0 disk.sda2.out‚0" memory_percent‚99.947190284729& proc.memory.availableMB‚ 8.65234375) proc.memory.percent‚20.73667049407959 cpu‚25.328423674138843) disk./.usagePercent‚78.65167175582337 disk.sda.in‚0 disk.sda.out‚ 169.36328125 disk.dm-1.in‚0 disk.dm-1.out‚ 169.36328125 disk.drbd1260.out‚0.796875 disk.dm-0.in‚ 12456.359375$ disk.dm-0.out‚147060.27685546875" disk.nvme0c0n1.in‚ 12456.359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 12456.359375 disk.sda3.in‚0# proc.memory.rssMB‚ 3397.49609375 proc.cpu.threads‚17 network.sent‚ 5.781694e+06% disk./.usageGB‚1171.9667472839355|×TÓ‚(:’ þÈèÔ–Îéà gpu.0.gpu‚63.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚251.395' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.27900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ÜüFxÔ‚(:í ÉèÔŽÊ¿ã disk.dm-1.out‚ 169.8359375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 12456.359375 disk.sda3.out‚ 169.8359375" disk.nvme0c0n1.in‚ 12456.359375) disk.nvme0c0n1.out‚147377.94091796875% disk./.usageGB‚1171.9432792663574 disk.drbd1260.out‚0.796875 disk.sda3.in‚0 disk.sda1.out‚0) proc.memory.percent‚20.73667049407959 cpu‚25.32845709660808 proc.cpu.threads‚17 disk.dm-1.in‚0$ disk.dm-0.out‚147377.94091796875' disk.nvme0n1.out‚147377.94091796875 disk.sda.in‚0 disk.sda.out‚ 169.8359375$ memory_percent‚99.94564056396484# proc.memory.availableMB‚8.90625 network.sent‚ 5.787075e+06 network.recv‚ 1.745434e+06) disk./.usagePercent‚78.65009679747273 disk.nvme0n1.in‚ 12456.359375 disk.sda1.in‚0# proc.memory.rssMB‚ 3397.49609375oÍ“Õ‚(:ˆ ÉèÔãÍÓã gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.599' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š­žÖ‚(:“ œÉèÔú‰¬à gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚43.0 gpu.0.powerWatts‚118.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/yd)lׂ(:á œÉèÔѾšâ disk.sda2.in‚0 disk.dm-0.in‚ 12456.359375 disk.dm-0.out‚147767.9296875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 12456.359375 disk.sda.in‚0 disk.sda.out‚ 170.2109375 disk.dm-1.in‚0 network.sent‚ 5.793045e+06 network.recv‚ 1.749893e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8046875 disk.nvme0n1.in‚ 12456.359375# disk.nvme0n1.out‚147767.9296875 disk.sda3.out‚ 170.2109375% disk.nvme0c0n1.out‚147767.9296875% disk./.usageGB‚1171.9667472839355 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 170.2109375# memory_percent‚99.9441146850586# proc.memory.rssMB‚ 3397.49609375 proc.cpu.threads‚17) disk./.usagePercent‚78.65167175582337# proc.memory.availableMB‚9.15625) proc.memory.percent‚20.73667049407959 cpu‚25.32848227218728–ë ÷2Ø- _wandbruntime‚5647  _runtime‚5647hp͆ÆÙÀ _runtime‚5647.217485141 _step‚163 & lr-AdamW/pg1‚0.0008469041089113308 & lr-AdamW/pg2‚0.0008469041089113308 trainer/global_step‚2049 $ _timestamp‚1.788486819675922e+09Ÿ¼Õ0ËÚÅ _runtime‚5647.217485141 _step‚163 & lr-AdamW/pg1‚0.0008469041089113308 & lr-AdamW/pg2‚0.0008469041089113308 trainer/global_step‚2049 $ _timestamp‚1.788486819675922e+09£âœ9½2Û- _wandbruntime‚5647  _runtime‚5647E^Ë1û'ÜõO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50778186 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚785 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670868434896384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚782 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚643141302848208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6243434430464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121894912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50752102 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11447362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6234903866272 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚636906398981936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42957909 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17556852 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126862063749632 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6243434430464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6231337028932 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93710011 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126862092847616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 trainer/global_step‚2049 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7991206400000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 % _timestamp‚1.7884868196773064e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42957915 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29003432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664628569296384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6239865600000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670868434896384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670872029154816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50778186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42957915 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664628594724352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670872029154816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93736101 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 _runtime‚5647.218881494 _step‚164 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42957909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29004217 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643137710582900 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50752102 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664628594724352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚636906373553968™èV(ÝúO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7991299196928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6239865600000 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50752102 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11447362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6234903866272 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚636906398981936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42957909 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17556852 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126862063749632 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6243434430464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6231337028932 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93710011 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126862092847616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 trainer/global_step‚2049 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7991206400000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 % _timestamp‚1.7884868196773064e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42957915 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29003432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664628569296384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6239865600000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670868434896384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670872029154816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50778186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42957915 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664628594724352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670872029154816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93736101 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 _runtime‚5647.218881494 _step‚164 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42957909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29004217 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643137710582900 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50752102 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664628594724352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚636906373553968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50778186 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚785 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670868434896384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚782 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚643141302848208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6243434430464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121894912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17564989 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7995143511040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7995050727424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚664943185740800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42978165 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚787 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11452849 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93780669 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17564205 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121881600 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚643445964358230 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 REDeviceStatsMonitor.on_train_batch_end/inaám-•×ctive_split.large_pool.freed‚11452846 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126923638723584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 trainer/global_step‚2049 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246429030400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29017051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134918782234624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134918660353024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6242860186624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6237894379094 _step‚165 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246429030400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50776394 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29017838 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93780669 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 _runtime‚5649.515898559 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671189614771200 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚664943160312832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6242860186624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6234327541546 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42978159 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637208044551168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42978165 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚08åtg 'àƒN C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93754553 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚671186020499456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50802504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788486821973417e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92783616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚643442372092714 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚637208069979136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17564989 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7995143511040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7995050727424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚664943185740800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42978165 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚787 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11452849 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93780669 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17564205 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121881600 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚643445964358230 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11452846 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126923638723584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 trainer/global_step‚2049 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246429030400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29017051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134918782234624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134918660353024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6242860186624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6237894379094 _step‚165 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246429030400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50776394 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29017838 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93780669 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 _runtime‚5649.515898559 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671189614771200 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚664943160312832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6242860186624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6234327541546 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42978159 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637208044551168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42978165 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.o@rý¶— n_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43970409 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51996138 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8183718090240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651989948841936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43970415 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129939386646016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651989923413968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686747742878208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29690218 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129939357548032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6393191589888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686747742878208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138122982841344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚680358145546752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6389622759424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136Y(ùúO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43970409 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51996138 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8183718090240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651989948841936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43970415 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129939386646016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651989923413968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686747742878208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29690218 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129939357548032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6393191589888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686747742878208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138122982841344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚680358145546752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6389622759424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680358145546752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95966553 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380889831640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚680358120118784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51970054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¨B`w2ú- _wandbruntime‚5767  _runtime‚5767)1Ù'ûþM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43990659 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚773 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680672736563200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52020456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43990659 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚652291594411168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96011121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43990665 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6383880344254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 $ _timestamp‚1.788486940018628e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138188513268224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6392617346048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6387447181802 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43990665 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29703124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121881600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51994346 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11727046 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680672736563200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130000903424000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658679067020938 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚652291619839136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 _runtime‚5767.560651117 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687065328481280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3F>ïúüƒN ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52020456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43990659 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚652291594411168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96011121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43990665 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 WHDeviceStatsMonit „(ùor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6383880344254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 $ _timestamp‚1.788486940018628e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138188513268224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6392617346048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6387447181802 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43990665 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29703124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121881600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51994346 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11727046 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680672736563200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130000903424000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658679067020938 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚652291619839136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 _runtime‚5767.560651117 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687065328481280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43990659 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚773 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680672736563200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136©¹¡2ý- _wandbruntime‚5767  _runtime‚5767º¢[þÕ 1loss/SRqqt4/detection_loss‚0.9455477965987876 /loss/FBt3/detection_loss‚2.7386504082844176 epoch‚3 /loss/FBt3/assignment_loss‚0.298262604615643 1loss/SRqqt3/detection_loss‚2.1989319368644997 0loss/FRt3/assignment_loss‚3.5301315397331394 /loss/FBt1/detection_loss‚2.6175385735653065 0loss/FRt2/assignment_loss‚2.6573369960452236 2loss/SRqqt2/assignment_loss‚1.5732194481151445 0loss/SRqqt2/detection_loss‚2.999352946451732 $loss/total_loss‚44.6149983495735 _runtime‚5767.562152387 1loss/SRqqt1/assignment_loss‚1.929678879181544 1loss/FBt2/assignment_loss‚0.09626725509729565 trainer/global_step‚2099 _step‚170 /loss/FBt2/detection_loss‚0.7564359763143302 /loss/FRt2/detection_loss‚2.0785750082347456 /loss/FRt1/assignment_loss‚3.860501000514397 0loss/FRt4/assignment_loss‚1.4484151751976313 2loss/SRqqt3/assignment_loss‚1.5769076850955133 2loss/SRqqt4/assignment_loss‚1.1206711754534067 0loss/FBt4/assignment_loss‚0.1265010639326647 /loss/FRt3/detection_loss‚2.2301824046010825 /loss/FBt4/detection_loss‚2.5739290795542975 $ _timestamp‚1.788486940020991e+09 1loss/FBt1/assignment_loss‚0.20547469663797444 /loss/FRt1/detection_loss‚2.9262890426012187 /loss/FRt4/detection_loss‚1.6365640415685383 1loss/SRqqt1/detection_loss‚2.4896336153149603Œb”¯`ÿÚ /loss/FRt3/detection_loss‚2.2301824046010825 /loss/FBt4/detection_loss‚2.5739290795542975 $ _timestamp‚1.788486940020991e+09 1loss/FBt1/assignment_loss‚0.20547469663797444 /loss/FRt1/detection_loss‚2.9262890426012187 /loss/FRt4/detection_loss‚1.6365640415685383 1loss/SRqqt1/detection_loss‚2.4896336153149603 1loss/SRqqt4/detection_loss‚0.9455477965987876 /loss/FBt3/detection_loss‚2.7386504082844176 epoch‚3 /loss/FBt3/assignment_loss‚0.298262604615643 1loss/SRqqt3/detection_loss‚2.1989319368644997 0loss/FRt3/assignment_loss‚3.5301315397331394 /loss/FBt1/detection_loss‚2.6175385735653065 0loss/FRt2/assignment_loss‚2.6573369960452236 2loss/SRqqt2/assignment_loss‚1.5732194481151445 0loss/SRqqt2/detection_loss‚2.999352946451732 $loss/total_loss‚44.6149983495735 _runtime‚5767.562152387 1loss/SRqqt1/assignment_loss‚1.929678879181544 1loss/FBt2/assignment_loss‚0.09626725509729565 trainer/global_step‚2099 _step‚170 /loss/FBt2/detection_loss‚0.7564359763143302 /loss/FRt2/detection_loss‚2.0785750082347456 /loss/FRt1/assignment_loss‚3.860501000514397 0loss/FRt4/assignment_loss‚1.4484151751976313 2loss/SRqqt3/assignment_loss‚1.5769076850955133 2loss/SRqqt4/assignment_loss‚1.1206711754534067 0loss/FBt4/assignment_loss‚0.1265010639326647ªè׃OU€‚(:Ê £ÊèÔ͘Äã disk.nvme0n1.in‚12705 disk.sda.out‚175.125 disk.dm-1.out‚175.125$ memory_percent‚99.94382858276367) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 disk.sda3.out‚175.125 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚150546.26611328125' disk.nvme0n1.out‚150546.26611328125 disk.sda3.in‚0 cpu‚25.32815447592569 network.sent‚ 5.923906e+06 network.recv‚ 1.786916e+06 disk.sda1.out‚0$ proc.memory.availableMB‚9.203125# proc.memory.rssMB‚ 3397.49609375( disk./.usagePercent‚78.7172785506156 disk.drbd1260.out‚ 0.8203125 disk.sda2.in‚0 disk.nvme0c0n1.in‚12705) disk.nvme0c0n1.out‚150546.26611328125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0" disk./.usageGB‚1172.9443359375 disk.dm-0.in‚12705Šë-b“‚(:ˆ £ÊèÔœ¥Ýã gpu.0.gpu‚74.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚319.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.9938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý‚o…‚‚(:ú ²ÊèÔïà¸á' disk.nvme0n1.out‚151033.57373046875 disk.sda.in‚0 disk.sda.out‚ 175.53515625 disk.dm-1.out‚ 175.53515625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.82421875 network.recv‚ 1.792359e+06% disk./.usageGB‚1172.9912719726562 disk.sda2.in‚0 disk.dm-0.in‚12906.62890625$ disk.dm-0.out‚151033.57373046875 disk.sda3.in‚0# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 disk.sda2.out‚0" disk.nvme0n1.in‚12906.62890625 disk.sda3.out‚ 175.53515625 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.328101314505624$ disk.nvme0c0n1.in‚12906.62890625) disk.nvme0c0n1.out‚151033.57373046875 disk.sda1.out‚0$ memory_percent‚99.94387626647949% proc.memory.availableMB‚ 9.1953125 proc.cpu.threads‚17 network.sent‚ 6.01029e+06) disk./.usagePercent‚78.72042846731689¿)QÕ“ƒ‚(:ˆ ²ÊèÔ‚€çâ gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚255.197' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êC„„‚(:ù ÁÊèÔ‹”‘ã disk.dm-1.in‚0 disk.drbd1260.out‚ 0.82421875 disk.sda2.out‚0 cpu‚25.328099406935184 proc.cpu.threads‚17 disk.sda.out‚ 175.8515625 disk.dm-1.out‚ 175.8515625$ memory_percent‚99.94385242462158& proc.memory.availableMB‚ 9.19921875# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 network.recv‚ 1.797241e+06) disk./.usagePercent‚78.72042846731689$ disk.nvme0c0n1.in‚12906.62890625) disk.nvme0c0n1.out‚151412.34716796875 disk.sda2.in‚0 disk.dm-0.in‚12906.62890625$ disk.dm-0.out‚151412.34716796875 disk.sda3.in‚0 network.sent‚ 6.018478e+06% disk./.usageGB‚1172.9912719726562 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚12906.62890625' disk.nvme0n1.out‚151412.34716796875 disk.sda3.out‚ 175.8515625 disk.sda.in‚0©»ïž…‚(:“ ÁÊèÔÙ²¤ã gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚221.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.242599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢î7xž†‚(:“ ÐÊèÔ³¤Ÿå gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚157.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.468200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¦»z‡‚(:ï ÐÊèÔé¹íå$ disk.nvme0c0n1.in‚12906.62890625( disk.nvme0c0n1.out‚151528.1494140625 disk.sda2.in‚0# proc.memory.rssMB‚ 3397.49609375) disk./.usagePercent‚78.72042846731689 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 176.2109375 disk.sda2.out‚0$ memory_percent‚99.94840621948242$ proc.memory.availableMB‚8.453125) proc.memory.percent‚20.73667049407959% disk./.usageGB‚1172.9912719726562 disk.drbd1260.in‚0 disk.drbd1260.out‚0.828125 disk.dm-0.in‚12906.62890625" disk.nvme0n1.in‚12906.62890625 disk.sda3.in‚0 cpu‚25.32808833665371 network.recv‚ 1.801633e+06 disk.sda.in‚0 disk.sda.out‚ 176.2109375 disk.sda1.in‚0# disk.dm-0.out‚151528.1494140625& disk.nvme0n1.out‚151528.1494140625 disk.sda3.out‚ 176.2109375 proc.cpu.threads‚17 network.sent‚ 6.0244e+06ÎÖ¨Qnˆ‚(:ã ßÊèÔ‡ë•â# proc.memory.rssMB‚ 3397.49609375 proc.cpu.threads‚17 network.sent‚ 6.027697e+06 network.recv‚ 1.805036e+06( disk.nvme0c0n1.out‚151774.7548828125 disk.dm-1.out‚176.875 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.3281351953173( disk./.usagePercent‚78.7204315394073 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚12906.62890625& disk.nvme0n1.out‚151774.7548828125% disk./.usageGB‚1172.9913177490234$ disk.nvme0c0n1.in‚12906.62890625 disk.drbd1260.out‚0.828125 disk.sda3.out‚176.875$ memory_percent‚99.94535446166992$ proc.memory.availableMB‚8.953125) proc.memory.percent‚20.73667049407959 disk.sda.in‚0 disk.sda.out‚176.875 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚151774.7548828125" disk.nvme0n1.in‚12906.62890625öﾓ‰‚(:ˆ ßÊèÔܧã gpu.0.gpu‚85.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚318.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.6818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t+ "ƒŠ‚(:ø îÊèÔôЫã disk.dm-1.in‚0 disk.drbd1260.out‚ 0.83203125 disk.sda3.out‚ 177.28515625 disk.sda1.in‚0 disk.dm-1.out‚ 177.28515625 disk.sda2.in‚0$ memory_percent‚99.94537830352783) proc.memory.percent‚20.73667049407959 cpu‚25.328095403107437 proc.cpu.threads‚17 network.sent‚ 6.032567e+06# disk.dm-0.out‚152190.6142578125& disk.nvme0n1.out‚152190.6142578125 disk.sda.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3397.49609375 network.recv‚ 1.808743e+06 disk.dm-0.in‚13082.84765625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚13082.84765625 disk.drbd1260.in‚0& proc.memory.availableMB‚ 8.94921875) disk./.usagePercent‚78.72043025936964$ disk./.usageGB‚1172.991298675537" disk.nvme0n1.in‚13082.84765625( disk.nvme0c0n1.out‚152190.6142578125 disk.sda.out‚ 177.28515625 disk.sda1.out‚0·Ôᾓ‹‚(:ˆ îÊèÔæåÒã gpu.0.gpu‚99.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚222.024' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.4048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤œr5Œ‚(:’ ýÊèÔóÄÉÝ gpu.0.gpu‚96.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚214.54' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.907999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}±‚!‚‚(:÷ ýÊèÔî¡Éß disk.dm-0.in‚13082.84765625# disk.dm-0.out‚152412.2236328125) proc.memory.percent‚20.73667049407959 cpu‚25.32805840604092 proc.cpu.threads‚17 network.sent‚ 6.038864e+06 network.recv‚ 1.813527e+06% disk./.usageGB‚1171.8334655761719( disk.nvme0c0n1.out‚152412.2236328125 disk.drbd1260.in‚0& disk.nvme0n1.out‚152412.2236328125 disk.sda3.out‚ 177.49609375# proc.memory.rssMB‚ 3397.49609375) disk./.usagePercent‚78.64272710858455 disk.sda.out‚ 177.49609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.83203125 disk.sda2.out‚0$ memory_percent‚99.94540214538574 disk.sda.in‚0 disk.dm-1.out‚ 177.49609375 disk.sda2.in‚0" disk.nvme0n1.in‚13082.84765625 disk.sda3.in‚0% proc.memory.availableMB‚ 8.9453125$ disk.nvme0c0n1.in‚13082.84765625êÐ>}Ž‚(:ò ŒËèÔß´ÒÞ disk.drbd1260.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚13082.84765625$ disk.nvme0n1.out‚152695.93359375 disk.sda3.in‚0$ memory_percent‚99.94843006134033 network.sent‚ 6.044944e+06 network.recv‚ 1.817939e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 178.03515625) proc.memory.percent‚20.73667049407959 cpu‚25.328037402676312 proc.cpu.threads‚17% disk./.usageGB‚1171.9220924377441 disk.sda3.out‚ 178.03515625& proc.memory.availableMB‚ 8.44921875# proc.memory.rssMB‚ 3397.49609375& disk.nvme0c0n1.out‚152695.93359375 disk.sda.out‚ 178.03515625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.8359375 disk.sda2.in‚0 disk.dm-0.in‚13082.84765625! disk.dm-0.out‚152695.93359375) disk./.usagePercent‚78.64867493162755$ disk.nvme0c0n1.in‚13082.84765625  IГ‚(:ˆ ŒËèÔÄËèÞ gpu.0.gpu‚73.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚227.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.4232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðôë 2- _wandbruntime‚5883  _runtime‚5883zÉ ‚Æ‘À trainer/global_step‚2149 $ _timestamp‚1.788487056383775e+09 _runtime‚5883.925280065 _step‚171 & lr-AdamW/pg1‚0.0008246347489776585 & lr-AdamW/pg2‚0.0008246347489776585@\¬–Ë’Å $ _timestamp‚1.788487056383775e+09 _runtime‚5883.925280065 _step‚171 & lr-AdamW/pg1‚0.0008246347489776585 & lr-AdamW/pg2‚0.0008246347489776585 trainer/global_step‚2149«¤ƒrF2“- _wandbruntime‚5883  _runtime‚5883tsSû'”õO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚702627050860032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6534009471688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 % _timestamp‚1.7884870563852503e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667073473273968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96991232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18381060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚801 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6530442634348 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53214090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98170915 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696087696369152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 _runtime‚5883.928020501 _step‚172 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚696087696369152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98197005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚702630645118464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44982915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8376244599296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚673603915908316 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚696087670941184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18381858 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44982909 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11995765 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚702627050860032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6542948749312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2149 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133016651346432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44982909 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696087670941184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30376822 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141392798954496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚673607508173624zІ}ô •úO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚696087670941184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18381858 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44982909 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11995765 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚702627050860032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6542948749312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2149 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133016651346432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44982909 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696087670941184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30376822 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141392798954496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚673607508173624 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚702627050860032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6534009471688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 % _timestamp‚1.7884870563852503e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667073473273968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96991232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18381060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚801 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6530442634348 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53214090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98170915 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696087696369152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 _runtime‚5883.928020501 _step‚172 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚696087696369152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98197005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚702630645118464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44982915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8376244599296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚673603915908316 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672¬¨>9Â2–- _wandbruntime‚5886  _runtime‚5886_AÚ'—ÿM WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98241573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673908577418130 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141458184194560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚702944636463104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8379986972160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667375169699136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45003159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133078197222400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2149 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6545943349248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30390494 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚781 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚696402261957632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53238408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141458308173312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6533433146962 % _timestamp‚1.7884870587060266e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 _runtime‚5886.248529018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45003165 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6545943349248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94880768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2149 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6545943349248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30390494 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚781 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚696402261957632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53238408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141458308173312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6533433146962 % _timestamp‚1.7884870587060266e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 _runtime‚5886.248529018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45003165 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6545943349248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94880768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98241573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673908577418130 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141458184194560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚702944636463104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8379986972160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667375169699136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45003159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133078197222400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 F‡hm§‚(:â æËèÔÕ¯Ðß disk.nvme0n1.in‚ 13261.875% proc.memory.availableMB‚ 8.6328125 network.recv‚ 1.842519e+06( disk.nvme0c0n1.out‚154207.4384765625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.84765625 disk.sda2.in‚0& disk.nvme0n1.out‚154207.4384765625 disk.sda3.out‚ 180.78515625# proc.memory.rssMB‚ 3397.49609375% disk./.usageGB‚1171.9869270324707 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0$ memory_percent‚99.94730949401855 cpu‚25.32785970995196 proc.cpu.threads‚17( disk./.usagePercent‚78.6530260356795 disk.nvme0c0n1.in‚13261.75 disk.dm-0.in‚13262# disk.dm-0.out‚154207.4384765625) proc.memory.percent‚20.73667049407959 network.sent‚ 6.157623e+06 disk.sda.in‚0 disk.sda.out‚ 180.78515625 disk.dm-1.in‚0 disk.dm-1.out‚ 180.78515625 disk.sda2.out‚0Η¼’~¨‚(:ó õËèÔÙ¸ˆß) disk./.usagePercent‚78.65302782773225 disk.dm-1.out‚ 181.1953125 disk.sda2.in‚0 disk.dm-0.in‚13402.54296875" disk.nvme0n1.in‚13402.54296875 disk.sda3.out‚ 181.1953125# memory_percent‚99.9457836151123% disk./.usageGB‚1171.9869537353516 disk.dm-1.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚154491.0791015625 disk.sda3.in‚0( disk.nvme0c0n1.out‚154491.0791015625% proc.memory.availableMB‚ 8.8828125 network.sent‚ 6.165938e+06 network.recv‚ 1.847426e+06 disk.sda.in‚0 disk.sda.out‚ 181.1953125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚13402.54296875 cpu‚25.32791918304088 disk.sda1.out‚0 disk.drbd1260.out‚ 0.84765625# disk.dm-0.out‚154490.2041015625# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 disk.sda2.out‚0²(á—ž©‚(:“ õËèÔæ”ñß gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚235.114' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.022800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ζޓª‚(:ˆ „ÌèÔê„àÝ gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚44.0 gpu.0.powerWatts‚143.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëry´«‚(:ö „ÌèÔ™©“à disk.drbd1260.in‚0) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚17 disk.dm-1.out‚ 181.6328125 disk.sda2.in‚0$ disk.dm-0.out‚154719.00146484375# proc.memory.rssMB‚ 3397.49609375 cpu‚25.32792311095432 network.recv‚ 1.851875e+06) disk./.usagePercent‚78.65332249240416" disk.nvme0n1.in‚13402.54296875' disk.nvme0n1.out‚154719.00146484375$ disk.nvme0c0n1.in‚13402.54296875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚154719.00146484375 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.8515625 disk.dm-0.in‚13402.54296875# memory_percent‚99.9457597732544& proc.memory.availableMB‚ 8.88671875 network.sent‚ 6.169881e+06% disk./.usageGB‚1171.9913444519043 disk.sda3.out‚ 181.6328125 disk.sda.out‚ 181.6328125B$£2¬- _wandbruntime‚6002  _runtime‚60029„^Å­¿ trainer/global_step‚2199 % _timestamp‚1.7884871751152668e+09 _runtime‚6002.656854653 _step‚175 % lr-AdamW/pg1‚0.000813097299647247 % lr-AdamW/pg2‚0.000813097299647247»+Sì樀 % lr-AdamW/pg2‚0.000813097299647247 trainer/global_step‚2199 % _timestamp‚1.7884871751152668e+09 _runtime‚6002.656854653 _step‚175 % lr-AdamW/pg1‚0.000813097299647247¯6Sœé2¯- _wandbruntime‚6002  _runtime‚6002Š:#uÿ'°ùO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12269965 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688837018571024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100427457 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54405958 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144662426822144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚136093974242816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚718506358841856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688840610836332 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45995409 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 % _timestamp‚1.7884871751166074e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚711817221763584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚711817221763584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6679995437056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718509953100288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45995415 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6683562274396 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718506358841856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31062889 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45995415 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6689137078272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144662552911360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6692705908736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚136093945144832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚711817247191552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚682157023133968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 trainer/global_step‚2199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96991232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚682157048561936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54432042 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718509953100288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54432042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18793722 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6692705908736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6002.661028955 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31063687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚795 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880%t‰!(±þO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100427457 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54405958 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144662426822144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚136093974242816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚718506358841856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688840610836332 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45995409 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 % _timestamp‚1.7884871751166074e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚711817221763584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚711817221763584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126089216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6679995437056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718509953100288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45995415 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6683562274396 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718506358841856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31062889 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45995415 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6689137078272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144662552911360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6692705908736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚136093945144832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚711817247191552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚682157023133968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 trainer/global_step‚2199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96991232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚682157048561936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54432042 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718509953100288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54432042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18793722 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6692705908736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6002.661028955 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31063687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚795 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12269965 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688837018571024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672°)C/b2²- _wandbruntime‚6004  _runtime‚6004Ò]î;³‚N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚712131838208000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚718827538716672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31076603 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54456360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18801157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6686552787218 XFDeviŠ <ÉÍceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6682985949670 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6004.850654688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718827538716672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚682458694131168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46015665 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6695700508672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12275449 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6695700508672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31077400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚797 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18801951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144727819475968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136155520118784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144727943454720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 trainer/global_step‚2199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54456360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46015665 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚689141680080838 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12275446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46015659 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6692131664896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 $ _timestamp‚1.788487177308948e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46015659 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682458719559136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100472025 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100445909 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚712131812780032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94880768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100445909 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚712131812780032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6692131664896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123978752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚689145272346354 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54456360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18801157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6686552787218 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6682985949670 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6004.850654688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718827538716672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚682458694131168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46015665 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6695700508672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12275449 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6695700508672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31077400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚797 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18801951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144727819475968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136155520118784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144727943454720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 trainer/global_step‚2199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54456360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46015665 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚689141680080838 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12275446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46015659 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6692131664896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 $ _timestamp‚1.788487177308948e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46015659 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682458719559136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100472025 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100445909 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚712131812780032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94880768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100445909 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚712131812780032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6692131664896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123978752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚689145272346354 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31076603 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 B%rÀ‚(:ç ÏÌèÔá²íß% disk./.usageGB‚1172.1667594909668 disk.drbd1260.out‚0.859375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3397.49609375 cpu‚25.327910602133496 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚20.73667049407959 network.sent‚ 6.280567e+06 network.recv‚ 1.873303e+06 disk.dm-0.in‚13530.76171875! disk.dm-0.out‚155941.00390625$ disk.nvme0n1.out‚155941.00390625$ disk.nvme0c0n1.in‚13530.76171875$ memory_percent‚99.94544982910156" proc.memory.availableMB‚8.9375) disk./.usagePercent‚78.66509474285866" disk.nvme0n1.in‚13530.76171875 disk.sda3.in‚0 disk.sda3.out‚ 183.890625& disk.nvme0c0n1.out‚155941.00390625 disk.sda.in‚0 disk.sda.out‚ 183.890625 disk.dm-1.out‚ 183.890625QjÖ•“Á‚(:ˆ ÏÌèÔ¸ð©ã gpu.0.gpu‚97.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚332.244' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Äïl‚(:á ÞÌèÔà¦Óá disk.sda1.in‚0 disk.sda3.out‚184.375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚184.375$ memory_percent‚99.94850158691406" proc.memory.availableMB‚8.4375 proc.cpu.threads‚17 network.sent‚ 6.283687e+06 network.recv‚ 1.876885e+06% disk./.usageGB‚1172.1901741027832% disk.nvme0n1.out‚156320.029296875 disk.sda3.in‚0 disk.sda.out‚184.375 disk.dm-1.in‚0# proc.memory.rssMB‚ 3397.49609375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚13530.76171875" disk.dm-0.out‚156320.029296875) proc.memory.percent‚20.73667049407959 cpu‚25.327867918203335) disk./.usagePercent‚78.66666611710383 disk.drbd1260.out‚0.859375" disk.nvme0n1.in‚13530.76171875$ disk.nvme0c0n1.in‚13530.76171875' disk.nvme0c0n1.out‚156320.029296875å#òí“Â(:ˆ ÞÌèÔ¾¨“â gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚249.341' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ú׊“Ä‚(:ˆ íÌèÔšÎáá gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚240.411' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ÑgnÅ‚(:ö íÌèÔûÊüá" disk.nvme0n1.in‚13681.40234375 disk.sda3.out‚ 184.71484375 proc.cpu.threads‚17) disk./.usagePercent‚78.68344382686779 disk.dm-1.out‚ 184.71484375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚13681.40234375 disk.sda3.in‚0$ memory_percent‚99.94847774505615# proc.memory.rssMB‚ 3397.49609375 network.sent‚ 6.289987e+06$ disk.nvme0c0n1.in‚13681.40234375' disk.nvme0c0n1.out‚156555.775390625" disk.dm-0.out‚156555.775390625& proc.memory.availableMB‚ 8.44140625) proc.memory.percent‚20.73667049407959 network.recv‚ 1.881719e+06 disk.sda.in‚0 disk.sda.out‚ 184.71484375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.86328125 disk.sda2.out‚0% disk.nvme0n1.out‚156555.775390625 cpu‚25.327838444294294% disk./.usageGB‚1172.4401741027832 disk.sda1.in‚0 disk.sda1.out‚0†H ‚Æ‚(:÷ üÌèÔŽø¼â network.sent‚ 6.295827e+06$ disk.dm-0.out‚156777.53271484375" disk.nvme0n1.in‚13681.40234375 disk.sda2.out‚0" proc.memory.availableMB‚8.6875# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 cpu‚25.327968248164694) disk./.usagePercent‚78.68186886851716 disk.sda3.out‚ 185.05078125$ disk.nvme0c0n1.in‚13681.40234375) disk.nvme0c0n1.out‚156777.53271484375 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1172.416706085205 disk.dm-0.in‚13681.40234375' disk.nvme0n1.out‚156777.53271484375 disk.dm-1.in‚0 disk.dm-1.out‚ 185.05078125 disk.drbd1260.in‚0 network.recv‚ 1.886447e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 185.05078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.86328125$ memory_percent‚99.94697570800781ºç¹€“Ç‚(:ˆ üÌèÔõÀ€ã gpu.0.gpu‚73.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚42.0 gpu.0.powerWatts‚117.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I:ŒÖ2È- _wandbruntime‚6122  _runtime‚6122nŽ.öÅÉ¿ % _timestamp‚1.7884872948146548e+09 _runtime‚6122.356228881 _step‚179 % lr-AdamW/pg1‚0.000801304107517978 % lr-AdamW/pg2‚0.000801304107517978 trainer/global_step‚22493¿"ÊÊÄ % lr-AdamW/pg1‚0.000801304107517978 % lr-AdamW/pg2‚0.000801304107517978 trainer/global_step‚2249 % _timestamp‚1.7884872948146548e+09 _runtime‚6122.356228881 _step‚179³=Mœ2Ë- _wandbruntime‚6122  _runtime‚6122=Ùêb ÌøO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚785 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102657909 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12544165 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19206442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139171268041216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚ 6122.35846153 trainer/global_step‚2249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31750607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94894080 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚727546798013952 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚727546798013952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6838894237696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚727546772585984 MGDeviceStatsMonitor.on_trainµ>øgï_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139171238943232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102631819 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12544162 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8760758648832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8760663754752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚697240572993968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6833115077104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55623910 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6842463068160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147931902697984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704073713499040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47007915 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734385666823680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123992064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147932026690048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47007915 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139171238943232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102631819 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12544162 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8760758648832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8760663754752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚697240572993968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6833115077104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55623910 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6842463068160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147931902697984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704073713499040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47007915 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734385666823680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123992064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147932026690048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47007915 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1725783552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102657909 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12544165 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19206442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139171268041216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚ 6122.35846153´Aìþ.2Î- _wandbruntime‚6124  _runtime‚6124ÿ Ù 'σN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102702477 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102702477 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31763451 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102676361 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚727861389030400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8764533896192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734703252426752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55648202 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697542243991168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6836105589926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147997318715392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚704374782743546 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6841888824320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102676361 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55674312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6845457668096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6841888824320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚727861363602432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12549649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19213805 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚704378375009062 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 _step‚181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55674312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47028165 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734706846698496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚727861363602432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139232784819200 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088iÛ¨°ÐˆN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55648202 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚734706846698496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31764234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12549646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102702477 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31763451 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102676361 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚727861389030400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8764533896192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734703252426752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55648202 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697542243991168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6836105589926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147997318715392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚704374782743546 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6841888824320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102676361 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55674312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6845457668096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6841888824320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚727861363602432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12549649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19213805 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚704378375009062 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 _step‚181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55674312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47028165 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734706846698496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚727861363602432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139232784819200 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102702477 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 F…ö*2ç- _wandbruntime‚6241  _runtime‚6241ôˆ—}ÿ'èùO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19619501 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚791 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743276323408384 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚743276348836352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6988651397120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56841862 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121894912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92796928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48020415 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚788 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48020409 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 trainer/global_step‚2299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6988651397120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104888361 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142248561839616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32437075 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6979101042472 _step‚184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48020415 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 _runtime‚6241.290140794 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719303223896440 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151201886304768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719306816161748 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6992220227584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6982667879812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56867946 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚743276348836352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚750264974805504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0®ßÜr éþO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚750264974805504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚25427968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32437866 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8953324465152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚712324122853968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151201764409856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8953231668224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12818365 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142248532741632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56867946 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884874137485006e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19618713 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121894912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92796928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48020415 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚788 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48020409 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 trainer/global_step‚2299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6988651397120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104888361 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142248561839616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32437075 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚345 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6979101042472 _step‚184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48020415 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 _runtime‚6241.290140794 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719303223896440 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151201886304768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719306816161748 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6992220227584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6982667879812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56867946 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚743276348836352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚750264974805504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19619501 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚791 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743276323408384 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚743276348836352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6988651397120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56841862 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884874159347248e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6991645983744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142310078617600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104932929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121881600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19626908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56866154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142310107715584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8957191016448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48040659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32450754 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚712625793851168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6985658392634 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚743590939852800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6995214827520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104906813 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750586154680320 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104932929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚743590914424832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚712625819279136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6995214827520 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750586154680320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48040659 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56866154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32451533 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚151267176850432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719611477671770 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719607885406254 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48040665 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174ÑS˜ 'ì‡N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750586154680320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48040659 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56866154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32451533 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚151267176850432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719611477671770 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719607885406254 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48040665 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104906813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6991645983744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92783616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884874159347248e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6991645983744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142310078617600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104932929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121881600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19626908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56866154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142310107715584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8957191016448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48040659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32450754 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚712625793851168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6985658392634 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚743590939852800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6995214827520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104906813 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750586154680320 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104932929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚743590914424832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚712625819279136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6995214827520 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968¹ç§j2í- _wandbruntime‚6243  _runtime‚6243rÊÈZîÔ 1loss/FBt2/assignment_loss‚0.19998592554416933 /loss/FBt4/detection_loss‚2.8385579586029053 trainer/global_step‚2299 % _timestamp‚1.7884874159372413e+09 2loss/SRqqt2/assignment_loss‚1.6578534457244372 2loss/SRqqt3/assignment_loss‚1.7253219043107137 /loss/FRt1/detection_loss‚2.9691143609859325 2loss/SRqqt1/assignment_loss‚1.5837846318525928 1loss/FBt1/assignment_loss‚0.31543186424435055 _runtime‚6243.478398929 .loss/FRt4/detection_loss‚1.779366209257801 /loss/FBt1/detection_loss‚2.7196015592279106 1loss/FBt3/assignment_loss‚0.31988194672618087 .loss/FRt2/detection_loss‚2.144766969713329 1loss/SRqqt3/detection_loss‚2.2578241604825724 _step‚186 epoch‚3 0loss/FBt4/assignment_loss‚0.5176375455443382 0loss/SRqqt2/detection_loss‚3.589624665285412 0loss/SRqqt4/detection_loss‚0.925314322655008 /loss/FBt2/detection_loss‚0.7338826021501669 /loss/FBt3/detection_loss‚2.3760777317425785 /loss/FRt1/assignment_loss‚4.407678798392967 /loss/FRt3/assignment_loss‚3.767585555712382 0loss/FRt4/assignment_loss‚1.8879219834544079 2loss/SRqqt4/assignment_loss‚1.1009297245182097 /loss/FRt3/detection_loss‚1.9484068246868755 %loss/total_loss‚47.05898187607362 0loss/SRqqt1/detection_loss‚2.629531058729911 /loss/FRt2/assignment_loss‚2.662900126528466Ò¨¼ž_ïÙ /loss/FBt4/detection_loss‚2.8385579586029053 trainer/global_step‚2299 % _timestamp‚1.7884874159372413e+09 2loss/SRqqt2/assignment_loss‚1.6578534457244372 2loss/SRqqt3/assignment_loss‚1.7253219043107137 /loss/FRt1/detection_loss‚2.9691143609859325 2loss/SRqqt1/assignment_loss‚1.5837846318525928 1loss/FBt1/assignment_loss‚0.31543186424435055 _runtime‚6243.478398929 .loss/FRt4/detection_loss‚1.779366209257801 /loss/FBt1/detection_loss‚2.7196015592279106 1loss/FBt3/assignment_loss‚0.31988194672618087 .loss/FRt2/detection_loss‚2.144766969713329 1loss/SRqqt3/detection_loss‚2.2578241604825724 _step‚186 epoch‚3 0loss/FBt4/assignment_loss‚0.5176375455443382 0loss/SRqqt2/detection_loss‚3.589624665285412 0loss/SRqqt4/detection_loss‚0.925314322655008 /loss/FBt2/detection_loss‚0.7338826021501669 /loss/FBt3/detection_loss‚2.3760777317425785 /loss/FRt1/assignment_loss‚4.407678798392967 /loss/FRt3/assignment_loss‚3.767585555712382 0loss/FRt4/assignment_loss‚1.8879219834544079 2loss/SRqqt4/assignment_loss‚1.1009297245182097 /loss/FRt3/detection_loss‚1.9484068246868755 %loss/total_loss‚47.05898187607362 0loss/SRqqt1/detection_loss‚2.629531058729911 /loss/FRt2/assignment_loss‚2.662900126528466 1loss/FBt2/assignment_loss‚0.19998592554416933º>¦!nð‚(:ã ƒÎèÔÿÉÑà disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚13914.00390625" disk.nvme0n1.in‚13914.00390625# disk.nvme0c0n1.out‚ 159762.21875 disk.sda.out‚ 188.9765625 disk.dm-1.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 159762.21875# memory_percent‚99.9464750289917) proc.memory.percent‚20.73667049407959 network.sent‚ 6.426062e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 188.9765625& proc.memory.availableMB‚ 8.76953125# proc.memory.rssMB‚ 3397.49609375 cpu‚25.32778186618427 network.recv‚ 1.924601e+06 disk.dm-1.out‚ 188.9765625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8828125 disk.dm-0.out‚ 159762.21875$ disk.nvme0c0n1.in‚13914.00390625 proc.cpu.threads‚17) disk./.usagePercent‚78.63363576902861$ disk./.usageGB‚1171.697998046875 disk.sda.in‚0ʳꙞñ‚(:“ ƒÎèÔâÅÆâ gpu.0.gpu‚98.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚46.0 gpu.0.powerWatts‚252.451' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.490199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]Ö¹Ü}ò‚(:ò ’ÎèÔô²ˆá disk.sda3.in‚0$ disk.nvme0c0n1.in‚13914.00390625 disk.sda.in‚0 disk.sda.out‚ 189.55078125 disk.dm-1.out‚ 189.55078125 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.930069e+06 disk.sda3.out‚ 189.55078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8828125 disk.dm-0.in‚13914.00390625! disk.dm-0.out‚160171.20703125# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959) disk./.usagePercent‚78.64590416208446$ disk.nvme0n1.out‚160171.20703125 disk.sda1.out‚0& proc.memory.availableMB‚ 8.51171875 cpu‚25.327712541033893 network.sent‚ 6.512274e+06% disk./.usageGB‚1171.8808059692383& disk.nvme0c0n1.out‚160171.20703125 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚99.94804859161377 proc.cpu.threads‚17" disk.nvme0n1.in‚13914.00390625ph•“ó‚(:ˆ ’ÎèÔÛŒÂá gpu.0.gpu‚98.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚250.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ëãÔÌô‚(:ø ¡ÎèÔ›¥–ß' disk.nvme0n1.out‚160320.96044921875 disk.sda3.out‚ 189.9140625$ disk.nvme0c0n1.in‚13914.00390625 disk.sda.out‚ 189.9140625 disk.drbd1260.out‚ 0.88671875# proc.memory.rssMB‚ 3397.49609375 disk.dm-0.in‚13914.00390625" disk.nvme0n1.in‚13914.00390625 disk.sda3.in‚0) disk.nvme0c0n1.out‚160320.96044921875 disk.dm-1.in‚0 disk.dm-1.out‚ 189.9140625 disk.drbd1260.in‚0 disk.sda2.in‚—Š·0 network.sent‚ 6.518536e+06 network.recv‚ 1.934958e+06$ disk.dm-0.out‚160320.96044921875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚99.94652271270752& proc.memory.availableMB‚ 8.76171875 cpu‚25.32766273789789) disk./.usagePercent‚78.64590416208446% disk./.usageGB‚1171.8808059692383 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚20.73667049407959 proc.cpu.threads‚177L “õ‚(:ˆ ¡ÎèÔÜâìá gpu.0.gpu‚47.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚50.0 gpu.0.powerWatts‚328.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.7492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$žJœö‚(:‘ °ÎèÔ¨Ç×á gpu.0.gpu‚99.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.78' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.35599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*ýr÷‚(:ç °ÎèÔ÷ííâ disk.dm-1.out‚190.375 disk.drbd1260.out‚ 0.88671875$ memory_percent‚99.94981288909912) proc.memory.percent‚20.73667049407959$ disk./.usageGB‚1171.880786895752 disk.sda2.in‚0' disk.nvme0n1.out‚160470.42138671875 disk.sda3.out‚190.375# disk.nvme0c0n1.in‚ 14042.0703125# proc.memory.rssMB‚ 3397.49609375( disk./.usagePercent‚78.6459028820468 disk.sda2.out‚0$ disk.dm-0.out‚160470.42138671875 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 8.22265625 cpu‚25.327528554907232 proc.cpu.threads‚17 network.sent‚ 6.52444e+06 network.recv‚ 1.939011e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚190.375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 14042.0703125! disk.nvme0n1.in‚ 14042.0703125) disk.nvme0c0n1.out‚160470.42138671875GÔöÜwø‚(:ì ¿ÎèÔ´Õéß# proc.memory.rssMB‚ 3397.49609375 disk.drbd1260.out‚0.890625 disk.dm-0.in‚ 14042.0703125 disk.sda3.in‚0) proc.memory.percent‚20.73667049407959( disk./.usagePercent‚78.6459059541372% disk./.usageGB‚1171.8808326721191 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 14042.0703125 cpu‚25.327488651104346 proc.cpu.threads‚17 network.sent‚ 6.529851e+06 network.recv‚ 1.942531e+06# disk.nvme0c0n1.in‚ 14042.0703125) disk.nvme0c0n1.out‚160813.60791015625 disk.sda.in‚0 disk.sda.out‚ 190.796875 disk.sda1.in‚0 disk.dm-1.out‚ 190.796875 disk.sda1.out‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚160813.60791015625' disk.nvme0n1.out‚160813.60791015625 disk.sda3.out‚ 190.796875$ memory_percent‚99.95079040527344" proc.memory.availableMB‚8.0625n¡€“ù‚(:ˆ ¿ÎèÔŠêá gpu.0.gpu‚99.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚49.0 gpu.0.powerWatts‚324.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.8174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ¨ÈW‚ú‚(:÷ ÎÎèÔ®îÊß disk.drbd1260.in‚0 network.sent‚ 6.536776e+06 disk.sda.out‚ 191.20703125 disk.sda1.out‚0! disk.nvme0n1.in‚ 14042.0703125) disk.nvme0c0n1.out‚161155.29150390625& proc.memory.availableMB‚ 8.51953125 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-0.in‚ 14042.0703125' disk.nvme0n1.out‚161155.29150390625 disk.sda3.in‚0 disk.sda3.out‚ 191.20703125$ memory_percent‚99.94800090789795 disk.drbd1260.out‚0.890625 network.recv‚ 1.945838e+06) disk./.usagePercent‚78.64590467409954% disk./.usageGB‚1171.8808135986328 disk.sda1.in‚0 disk.dm-1.in‚0$ disk.dm-0.out‚161155.29150390625# disk.nvme0c0n1.in‚ 14042.0703125 disk.dm-1.out‚ 191.20703125 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3397.49609375) proc.memory.percent‚20.73667049407959 cpu‚25.327426007305437w¿³ö“û‚(:ˆ ÎÎèÔέ¦á gpu.0.gpu‚97.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚47.0 gpu.0.powerWatts‚251.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[œO·“ü‚(:ˆ ÝÎèÔòòÝà gpu.0.gpu‚57.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚68.11531066894531* gpu.0.memoryAllocatedBytes‚ 58510606336 gpu.0.temp‚48.0 gpu.0.powerWatts‚312.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚62.5722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÖ:Å€ý‚(:õ ÝÎèÔÊ܈â$ memory_percent‚99.94957447052002 network.sent‚ 6.541122e+06( disk.nvme0c0n1.out‚161473.2138671875 disk.sda.out‚ 191.94921875# proc.memory.rssMB‚ 3397.49609375 cpu‚25.32745847096238 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 14042.0703125# disk.dm-0.out‚161473.2138671875& proc.memory.availableMB‚ 8.26171875) proc.memory.percent‚20.73667049407959 network.recv‚ 1.950606e+06) disk./.usagePercent‚78.64590467409954% disk./.usageGB‚1171.8808135986328# disk.nvme0c0n1.in‚ 14042.0703125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 191.94921875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.89453125! disk.nvme0n1.in‚ 14042.0703125& disk.nvme0n1.out‚161473.2138671875 disk.sda3.in‚0 disk.sda3.out‚ 191.94921875V”ýs2þ- _wandbruntime‚6356  _runtime‚6356ÐC )ÇÿÁ trainer/global_step‚2349 % _timestamp‚1.7884875294411025e+09 _runtime‚6356.984402653 _step‚187 & lr-AdamW/pg1‚0.0007769900861808629 & lr-AdamW/pg2‚0.0007769900861808629Ê!ñˆÌ€Æ & lr-AdamW/pg1‚0.0007769900861808629 & lr-AdamW/pg2‚0.0007769900861808629 trainer/global_step‚2349 % _timestamp‚1.7884875294411025e+09 _runtime‚6356.984402653 _step‚187»žÍˆ2- _wandbruntime‚6356  _runtime‚6356n²ºmÿ'‚ùO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7138408556544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145325855638016 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58059814 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚766144282787328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚766144282787328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58059814 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33124165 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7128653845180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7141977387008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734539918824456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154472341320704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759005899658752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7141977387008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119797760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90699776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49032909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚758 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13092562 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9146394982912 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58085898 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚761 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13092565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20032361 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33124926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20031603 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145325826540032 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚766147877045760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 trainer/global_step‚2349 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49032909 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9146485682688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107118813 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759005874230784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 % _timestamp‚1.7884875294422116e+09¤3O(ƒþO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚29097984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 trainer/global_step‚2349 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49032909 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9146485682688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107118813 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759005874230784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 % _timestamp‚1.7884875294422116e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7138408556544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145325855638016 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566837340 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚3136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58059814 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚766144282787328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚766144282787328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457466880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58059814 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33124165 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚3136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7128653845180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7141977387008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 793275904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734539918824456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154472341320704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759005899658752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7141977387008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119797760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90699776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49032909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚758 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568830464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13092562 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9146394982912 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 53397684224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58085898 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457466880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚761 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13092565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20032361 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33124926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20031603 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145325826540032 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚766147877045760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 53537329673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0¼²9—»2„- _wandbruntime‚6359  _runtime‚6359b\SjÌ…ƒN NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58084106 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455452088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚766465462662144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3136 % _timestamp‚1.7884875317396612e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145387401513984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734844580334478 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7131644357794 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4559208448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1319 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 _step‚189 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107163381 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚727709369139136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20039929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88589312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145387372416000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33137975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13098046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734840988068962 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759320465247232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7144971986944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7135211195342 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7144971986944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49053159 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9150287211520 4Ÿø•=F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9150375800832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58084106 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154537659627520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚727709343711168 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107163381 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49053165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49053159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚755 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6359.282338258 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚766461868390400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33138730 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117687296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107137265 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107137265 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7141403143168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7141403143168 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154537777314816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚766465462662144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759320490675200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58110216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49053165 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107163381 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚727709369139136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3886 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20039929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88589312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145387372416000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2174 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33137975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457466880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13098046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568843776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734840988068962 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759320465247232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7144971986944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7135211195342 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7144971986944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 57956892672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49053159 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9150287211520 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9150375800832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58084106 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154537659627520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚727709343711168 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107163381 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 52238108672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49053165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49053159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚755 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 793275904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6359.282338258 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚766461868390400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33138730 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚345 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117687296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107137265 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107137265 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7141403143168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7141403143168 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154537777314816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 56018903552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚766465462662144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2174 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759320490675200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58110216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49053165 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 trainer/global_step‚2399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13366765 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚749773021487164 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚742491222573968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7288165715968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚148403120338432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚782027185027584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚782023590769152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774735450481152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 _step‚192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7291734546432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7288165715968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7291734546432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33810629 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109323175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33811367 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50045415 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9340128668160 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 % _timestamp‚1.7884876464451253e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50045415 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚782023590769152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59277766 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20444602 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7281773485228 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 _runtime‚6473.986796819 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚735 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157743162501120 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59277766 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86505472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088ŠœN(ŸþO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 52238108672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 53397684224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7291734546432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33810629 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5310 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3886 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2174 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109323175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33811367 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 986103808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 57956892672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50045415 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9340128668160 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3136 % _timestamp‚1.7884876464451253e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50045415 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4559208448 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2174 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 56018903552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚782023590769152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568830464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59277766 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20444602 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7281773485228 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4559208448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594258432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚5310 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2174 _runtime‚6473.986796819 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚735 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157743162501120 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3886 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 52238108672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4559208448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 57956892672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59277766 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86505472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455452088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 53397684224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5310 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148403149436416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 57956892672 trainer/global_step‚2399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594258432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13366765 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚749773021487164 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚742491222573968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 49728133120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7288165715968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚148403120338432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚782027185027584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚782023590769152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774735450481152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3592265308 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 56018903552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5310 _step‚192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7291734546432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7288165715968À .[2 - _wandbruntime‚6476  _runtime‚6476>D– '¡ƒN NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59302058 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚782341176372224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 _runtime‚6476.168517612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7291160302592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13372246 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20452998 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148464666214400 % _timestamp‚1.7884876486268787e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109393833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50065665 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7294729146368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157808693426688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9343909524992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109367717 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109367717 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59328168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33824487 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚742792918999136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7281197160502 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157808575739392 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 trainer/global_step‚2399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7294729146368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚775050016069632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚750077682997186 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚750074090731670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117687296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50065665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0=…ò‡<¢ˆN NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59328168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚775050041497600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚782344770643968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1725783552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 56018903552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3592265516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 49728133120 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566837548 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚5310 _runtime‚6476.168517612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4559208448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457466880 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 53537329673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4559208448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7291160302592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13372246 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20452998 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148464666214400 % _timestamp‚1.7884876486268787e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594271744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109393833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 986103808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 52238108672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 53397684224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5310 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50065665 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594271744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7294729146368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157808693426688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9343909524992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109367717 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109367717 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59328168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 53397684224 K>DeviceStatsMÑóŸÆÒ onitor.on_train_batch_end/inactive_split.all.freed‚33824487 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568843776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚742792918999136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7281197160502 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157808575739392 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3136 trainer/global_step‚2399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7294729146368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚775050016069632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚750077682997186 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 57956892672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚29097984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚750074090731670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117687296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 57956892672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 53397684224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50065665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59302058 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚782341176372224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚5310 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43386250734329224 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991831362247467 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 *loss/val_total_loss‚47.305956164594605 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9975894093513489 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 jet/accuracy_6_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.021767565980553627 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21364428102970123 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 jet/accuracy_3_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13873447477817535 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.19839903712272644 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15194013714790344 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5544585585594177 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2494908571243286 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16166582703590393 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912924766540527 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904401898384094 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.38633400201797485 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2178667187690735 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04285714402794838 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9772727489471436 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3635522425174713 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40878885984420776 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.611111044883728 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3909701108932495 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4551851451396942 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.21666666865348816 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.3947368562221527 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5192307829856873 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5666666626930237 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005019719712436199 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.10000000149011612 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.59375 particle/accuracy_1_of_2‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2877110242843628 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666567325592 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5463414788246155 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3645471930503845 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14369112253189087 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8088924288749695 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,jet/accuracy_3_of_3‚0.021389180794358253 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08620689809322357 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5161290168762207 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 'particle/accuracy‚0.812337338924408 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004608294926583767 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24997007846832275 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29639559984207153 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9838144779205322 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9943825602531433 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1607998162508011 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013112017884850502 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20216450095176697 *jet/accuracy_2_of_3‚0.2236449271440506 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00878656841814518 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.31843435764312744 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40554288029670715 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974682331085205 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.04833333566784859 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.32434922456741333 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995699524879456 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41751977801322937 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16689057648181915 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31419846415519714 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925565123558044 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03481481596827507 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/1FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2222222238779068 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984114766120911 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09948679804801941 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2926587462425232 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9822348952293396 7 Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6071428656578064 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.49064627289772034 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0010752688394859433 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.300403038039804e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21379075944423676 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4236110746860504 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9583333730697632 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9774298071861267 , Purity/1FRt3SRqqt*FBt/FBt_purity‚0.96875 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_1_of_5‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.917658805847168 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.29851627349853516 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003150201519019902 -"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4375 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014461373910307884 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12091503292322159 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.37752363085746765 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3331637978553772 ?&Purity/1FRt*SRqqt2FBt/event_proportion‚0.022303426638245583 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.04731602966785431 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5469791293144226 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023591697216033936 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_4_of_4‚1 ?&Purity/0FRt1SRqqt2FBt/event_proportion‚0.008757560513913631 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44769933819770813 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9224139451980591 @&Purity/*FRt*SRqqt4FBt/event_proportion‚0.0003780241822823882 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12942346930503845 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16231097280979156 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1287122517824173 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5929723381996155 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 >&Purity/0FRt*SRqqt1FBt/event_proportion‚0.14787046611309052 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3212536573410034 >&Purity/2FRt1SRqqt*FBt/event_proportion‚0.04158266261219978 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.300403038039804e-05 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4947892129421234 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21557186543941498 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011947430903092027 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.004032257944345474 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ?&Purity/*FRt2SRqqt2FBt/event_proportion‚0.002079133177176118 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14969196915626526 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013230846961960196 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006111391354352236 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9591733813285828 ?&Purity/*FRt1SRqqt2FBt/event_proportion‚0.017011089250445366 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.392173171043396 particle/accuracy_1_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6190475821495056 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.22886277735233307 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.029674898833036423 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2481723576784134 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0483240932226181 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9814197421073914 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.43580374121665955 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3896947503089905 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3804347813129425 :"Purity/0FRt0SRqqt3FBt/event_purity‚0.12999999523162842 @&Purity/1FRt4SRqqt0FBt/event_proportion‚0.0003780241822823882 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9931899905204773 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6349245309829712 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9939655065536499 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37622693181037903 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015605362132191658 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.491106241941452 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2168874591588974 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3833892345428467 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.6160534024238586 ?&Purity/*FRt0SRqqt3FBt/event_proportion‚0.003717237850651145 >&Purity/0FRt0SRqqt1FBt/event_proportion‚0.08606351166963577 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9935089349746704 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35681816935539246 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0025201612152159214 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18105420470237732 >&Purity/*FRt0SRqqt1FBt/event_proportion‚0.14591734111309052 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3010340631008148 ?&Purity/0FRt3SRqqt*FBt/event_proportion‚0.006741431541740894 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.01315789483487606 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18865740299224854 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.300403038039804e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15754950046539307 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.300403038039804e-05 +particle/sensitivity‚0.4377148151397705 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15763840079307556 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006111391354352236 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3589743971824646 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1527632623910904 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005670362734235823 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1612127423286438 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332535296678543 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1989724338054657 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20168863236904144 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08688529580831528 ,&Purity/0FRt3SRq;+ö„Oqt2FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4020936191082001 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚ 0.1953125 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5256833434104919 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/0FRt2SRqqt0FBt/event_proportion‚0.043283771723508835 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3697267174720764 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0027091733645647764 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.36666667461395264 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_3_of_5‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5524136424064636 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1723790317773819 @&Purity/2FRt3SRqqt1FBt/event_proportion‚0.0003780241822823882 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666567325592 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13988357782363892 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15677274763584137 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.300403038039804e-05 +jet/accuracy_3_of_4‚0.06108808517456055 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6190475821495056 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5214523077011108 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.411786824464798 ?&Purity/2FRt1SRqqt2FBt/event_proportion‚0.001890120911411941 +jet/accuracy_2_of_4‚0.33747735619544983 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006048386916518211 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5039336681365967 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987964630126953 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9244444966316223 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09929435700178146 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0024571572430431843 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.30420443415641785 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558467626571655 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4607142508029938 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4402777850627899 *jet/accuracy_3_of_5‚0.0770573616027832 particle/accuracy_1_of_4‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006300403038039804 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008064515888690948 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00025201612152159214 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt*FBt/event_proportion‚0.11857359111309052 ?&Purity/2FRt0SRqqt0FBt/event_proportion‚0.024571573361754417 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ?&Purity/*FRt*SRqqt3FBt/event_proportion‚0.004473286215215921 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09567901492118835 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5451740622520447 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46369048953056335 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14722223579883575 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ?&Purity/2FRt*SRqqt2FBt/event_proportion‚0.005418346729129553 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19214698672294617 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4082483649253845 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36989420652389526 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4555555582046509 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3460533618927002 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9197531938552856 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993280172348022 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4574206471443176 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3811022639274597 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5276785492897034 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.16981567442417145 7 Purity/1FRt3SRqqt1FBt/FRt_purity‚0.2888889014720917 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9868420958518982 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.38905349373817444 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚ 0.1953125 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9858958125114441 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928581118583679 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9689620137214661 >&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19745463132858276 ?&Purity/1FRt2SRqqt1FBt/event_proportion‚0.008946572430431843 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014064434915781021 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.01904761977493763 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015435988083481789 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 trainer/global_step‚2415 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9826490879058838 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20640520751476288 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.37933674454689026 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/0FRt*SRqqt0FBt/event_proportion‚0.21931703388690948 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4061177968978882 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00025201612152159214 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.41743406653404236 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.300403038039804e-05 6 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.347222238779068 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5131515264511108 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.06747731566429138 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5346819162368774 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0982142835855484 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7982456088066101 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22727273404598236 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.98328697681427 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9811828136444092 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5277777910232544 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007487853057682514 +jet/accuracy_3_of_6‚0.10526315867900848 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9982081055641174 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19083820283412933 -jet/accuracy_5_of_5‚0.0026881720405071974 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03542044758796692 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06982892006635666 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15224334597587585 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15260526537895203 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11420890688896179 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 particle/accuracy_6_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt2SRqqt1FBt/event_proportion‚0.02255544438958168 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9824560880661011 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9254385232925415 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016562648117542267 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 _runtime‚6573.296434916 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02973990887403488 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003150201519019902 jet/accuracy_1_of_7‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17718957364559174 ?&Purity/2FRt2SRqqt1FBt/event_proportion‚0.003969253972172737 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024571572430431843 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09305164217948914 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1669294536113739 ?&Purity/3FRt1SRqqt*FBt/event_proportion‚0.003717237850651145 *jet/accuracy_1_of_2‚0.6879146695137024 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt*SRqqt*FBt/event_proportion‚0.40996724367141724 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8042209148406982 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012600806076079607 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt*FBt/event_proportion‚0.10244455933570862 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880530834197998 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012600806076079607 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.31245461106300354 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00025201612152159214 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907549619674683 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3589743375778198 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24444444477558136 >&Purity/0FRt*SRqqt2FBt/event_proportion‚0.03944052383303642 A&Purity/0FRt1SRqqt3FBt/event_proportion‚0.00044102821266278625 -"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5625 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1367940604686737 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15224334597587585 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.36424294114112854 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012600806076079607 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0010710685746744275 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4482736885547638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9855555295944214 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007875503972172737 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.175428569316864 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0003780241822823882 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.008333333767950535 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890903830528259 +jet/accuracy_1_of_1‚0.41793009638786316 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 6 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.988095223903656 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16646477580070496 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003150201519019902 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4029024839401245 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01486289594322443 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9988052845001221 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48315152525901794 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3506714999675751 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3103686571121216 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3706011176109314 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6938586235046387 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006930443341843784 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01827116869390011 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3546414077281952 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3759216070175171 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15785019099712372 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_3_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9444444179534912 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.362579345703125 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45394736528396606 (particle/f_score‚0.42144331336021423 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00025201612152159214 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.39794373512268066 @&Purity/1FRt4SRqqt*FBt/event_proportion‚0.0003780241822823882 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15597887337207794 particle/accuracy_4_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9841269850730896 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.300403038039804e-05 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5118893384933472 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1001134067773819 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3635522425174713 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5740646123886108 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5443276762962341 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39838674664497375 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34777331352233887 ?&Purity/0FRt2SRqqt1FBt/event_proportion‚0.009261592291295528 >&Purity/*FRt1SRqqt0FBt/event_proportion‚0.33039313554763794 ?&Purity/1FRt3SRqqt0FBt/event_proportion‚0.007938507944345474 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0026881720405071974 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 % _timestamp‚1.7884877457525837e+09 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0015120967291295528 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22736623883247375 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9273655414581299 jet/accuracy_4_of_6‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3555998206138611 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.37501558661460876 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 epoch‚3 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016237638890743256 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021722603123635054 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.300403038039804e-05 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993280172348022 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 jet/accuracy_2_of_7‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.43598270416259766 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4022810161113739 +particle/specificity‚0.8816567659378052 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 particle/accuracy_7_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.980875551700592 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 particle/accuracy_2_of_4‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7807018160820007 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.966666579246521 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9815475344657898 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.75 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.261904776096344 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.261904776096344 _step‚195 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_5_of_5‚1 particle/accuracy_2_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ."Purity/1FRt*SRqqt3FBt/event_purity‚0.09375 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8624902367591858 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15503843128681183 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3011619746685028 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816ÃÁɧn0¶‡§ +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 ?&Purity/2FRt0SRqqt1FBt/event_proportion‚0.008946572430431843 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993280172348022 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0005040322430431843 jet/accuracy_2_of_7‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3135710656642914 >&Purity/1FRt1SRqqt1FBt/event_proportion‚0.03944052383303642 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.43598270416259766 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4022810161113739 +particle/specificity‚0.8816567659378052 >&Purity/*FRt*SRqqt1FBt/event_proportion‚0.27079132199287415 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 particle/accuracy_7_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.980875551700592 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt1FBt/event_proportion‚0.0011340725468471646 particle/accuracy_2_of_4‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7807018160820007 @&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0028351815417408943 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.966666579246521 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003150201519019902 ?&Purity/1FRt0SRqqt2FBt/event_proportion‚0.015120967291295528 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9815475344657898 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.019468246027827263 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.75 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.261904776096344 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013230846961960196 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.261904776096344 _step‚195 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.300403038039804e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.300403038039804e-05 particle/accuracy_5_of_5‚1 particle/accuracy_2_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ."Purity/1FRt*SRqqt3FBt/event_purity‚0.09375 >&Purity/1FRt0SRqqt0FBt/event_proportion‚0.13142640888690948 >&Purity/2FRt1SRqqt0FBt/event_proportion‚0.03181703761219978 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8624902367591858 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15503843128681183 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3011619746685028 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9902653694152832 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.17874395847320557 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21713504195213318 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24497175216674805 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9833081364631653 >&Purity/*FRt2SRqqt0FBt/event_proportion‚0.10855595022439957 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43386250734329224 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991831362247467 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.049773186445236206 *loss/val_total_loss‚47.305956164594605 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9975894093513489 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt0SRqqt*FBt/event_proportion‚0.037046369165182114 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9964157938957214 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012600806076079607 jet/accuracy_6_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt0SRqqt0FBt/event_proportion‚0.15757308900356293 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04082661122083664 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.021767565980553627 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21364428102970123 @&Purity/3FRt*SRqqt0FBt/event_proportion‚0.0071824598126113415 jet/accuracy_3_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13873447477817535 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.19839903712272644 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15194013714790344 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5544585585594177 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2494908571243286 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16166582703590393 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912924766540527 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904401898384094 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.38633400201797485 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2305317521095276 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2178667187690735 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04285714402794838 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9772727489471436 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3635522425174713 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011340725468471646 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40878885984420776 >&Purity/2FRt*SRqqt0FBt/event_proportion‚0.07579384744167328 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.611111044883728 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3909701108932495 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4551851451396942 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.05361643061041832 8 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.21666666865348816 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.3947368562221527 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5192307829856873 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5666666626930237 ?&Purity/0FRt0SRqqt3FBt/event_proportion‚0.002583165420219302 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4366179406642914 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005019719712436199 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.10000000149011612 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 , Purity/1FRt*SRqqt3FBt/FRt_purity‚0.59375 particle/accuracy_1_of_2‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2877110242843628 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666567325592 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5463414788246155 ?&Purity/2FRt3SRqqt0FBt/event_proportion‚0.003654233878478408 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.300403038039804e-05 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3645471930503845 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14369112253189087 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1265120953321457 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8088924288749695 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002142137149348855 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,jet/accuracy_3_of_3‚0.021389180794358253 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08620689809322357 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt*FBt/event_proportion‚0.009954636916518211 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009450604557059705 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5161290168762207 @&Purity/4FRt*SRqqt0FBt/event_proportion‚0.0001890120911411941 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 'particle/accuracy‚0.812337338924408 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004608294926583767 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24997007846832275 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29639559984207153 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt*FBt/event_proportion‚0.020476309582591057 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9838144779205322 A&Purity/0FRt4SRqqt0FBt/event_proportion‚0.00044102821266278625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9943825602531433 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1607998162508011 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013112017884850502 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0030871976632624865 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20216450095176697 *jet/accuracy_2_of_3‚0.2236449271440506 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0010080644860863686 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.300403038039804e-05 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00878656841814518 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.31843435764312744 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40554288029670715 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974682331085205 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.009135584346950054 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.04833333566784859 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.32434922456741333 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995699524879456 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41751977801322937 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16689057648181915 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31419846415519714 ?&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021169355139136314 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt1SRqqt1FBt/event_proportion‚0.05185231938958168 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925565123558044 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1694808453321457 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.05727066472172737 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03481481596827507 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/1FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ?&Purity/2FRt0SRqqt2FBt/event_proportion‚0.003465221729129553 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005670362734235823 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2222222238779068 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984114766120911 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.0994867